Computer speech recognition and semantic understanding from activity patterns

ABSTRACT

A user activity pattern may be ascertained using signal data from a set of computing devices. The activity pattern may be used to infer user intent with regards to a user interaction with a computing device or to predict a likely future action by the user. In one implementation, a set of computing devices is monitored to detect user activities using sensors associated with the computing devices. Activity features associated with the detected user activities are determined and used to identify an activity pattern based on a plurality of user activities having similar features. Examples of user activity patterns may include patterns based on time, location, content, or other context. The inferred user intent or predicted future actions may be used to facilitate understanding user speech or determining a semantic understanding of the user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application Claims the benefit of prior filed applications. Thisapplication is a continuation of U.S. application Ser. No. 15/087,224,titled “Computer Speech Recognition And Semantic Understanding FromActivity Patterns,” filed Mar. 31, 2016, which is a continuation-in-partapplication of U.S. application Ser. No. 14/941,155, titled “EnhancedComputer Experience From Activity Prediction,” filed Nov. 13, 2015, eachof which is hereby expressly incorporated by reference in its entirety.

BACKGROUND

People are increasingly interacting with computing devices and relyingon these devices for information, recommendations, and other services toassist them in their day-to-day tasks. But understanding the user'sspoken words and intent by the computing device in these interactionsremains a difficult technical problem. In such interactions, users areoften left frustrated by the inability of their computerized personalassistant applications or services to understand them, their intent, oranticipate their needs.

At the same time, many users of computing devices have repeatingpatterns of usage. For example, a user may launch an email app on theirmobile device every workday morning, before starting work, browse to afavorite news website over lunchtime, call a close friend or familymember on their drive home from work, or use their laptop computer toplan their annual summer vacation around May. By learning these patternsof user activity, the computerized personal-assistant applications andservices can provide improved user experiences including improvedunderstanding of the user's speech and the user's intent.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used in isolation as an aid in determining the scope of the claimedsubject matter.

Embodiments described in the present disclosure are directed towardsunderstanding user speech and inferring user intent regarding userinteractions performed via the user's computing devices. In particular,embodiments may determine a likely spoken utterance or intent of a userwith regards to an interaction with a computing device, or may predict alikely future action to be performed by a user, based on a history ofsensed user activity. Data corresponding to user activity may begathered over time using sensors of one or more of the user's computingdevices. From this historical user activity information, a computersystem may learn user activity patterns associated with the computingdevices. By analyzing a user activity pattern, future user actions maybe predicted or user intent may be inferred.

For example, computing devices associated with a user (“user devices”)may employ one or more sensors to generate data relevant to a user'sactivity on a user device(s). The user activity may be monitored,tracked, and used for determining patterns of user activity. Examples ofuser activity patterns may include, without limitation, activitypatterns based on time, location, content, or other context, asdescribed herein. In some embodiments, the activity patterns may bedetermined based on user activity related to browsing, applicationusage, or other related user activities associated with one or more userdevices or user activity otherwise determinable via the one or more userdevices.

Based on the determined patterns of user activity, user intent may beinferred regarding user interactions with a computing device, and/orpredictions of user activity determined and used to provide improveduser experiences. Examples of improved user experiences, which arefurther described herein, may include improved speech recognition orimproved semantic understanding of the user, which may be used for moreaccurately resolving, disambiguating, and/or understand a user's speech,or other aspects of understanding input from the user and relatedenhanced computer experiences such as personalization. In someembodiments, the user activity patterns, or the inferred intent orpredictions of future activity determined therefrom, may be madeavailable to one or more applications and services that consume thisinformation to provide an improved user experience. Some embodiments maybe incorporated into (or operate in conjunction with) an automaticspeech recognition (ASR) and/or language modeling component or service.In this manner, operating in conjunction with (or as a component of) thespeech recognition and interpretation operation, the user's utterancemay be resolved in such a manner as to be consistent with learnedpatterns of user activity.

Accordingly, as will be further described, in one embodiment, one ormore user devices associated with a user may be identified and monitoredfor user activity. In some embodiments user activity monitoring may befacilitated using an application or service that runs on the monitoreduser device. User actions or “activity events”, such as visiting awebsite, launching an application, or other actions similar to thosedescribed herein, may be detected and logged with associated contextualdata; for example, by logging the observed user action with a timestamp,location stamp, and/or associating the activity event with otheravailable contextual information. From the logged user activity,historical user activity information may be determined and provided toan inference engine. Based on an analysis of historical user activity, auser activity pattern may be determined. In some embodiments, a semanticanalysis is performed on the activity events and associated informationto characterize aspects of the activity event. The semantic analysis maybe used to characterize information associated with the activity eventand may provide other relevant features of the activity events that maybe used to identify patterns.

The user activity patterns may be used to infer user intent or predictfuture activity of the user. From these predictions or inferred userintent, various implementations may provide personalized computingexperiences and other services tailored to the user, such as improvedspeech recognition; incorporation of the user's routine intorecommendations and notifications; improved semantic understanding bythe user's computer devices, or other examples as described herein. Inthis way, embodiments of the disclosure improve the operation of thecomputing device and thus provide more accurate speech recognition,improved resolution and disambiguation, a more accurate understanding ofthe user's intent, and thus an improved user experience with thecomputing device.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the disclosure are described in detail below with referenceto the attached drawing figures, wherein:

FIG. 1 is a block diagram of an example operating environment suitablefor implementations of the present disclosure;

FIG. 2 is a diagram depicting an example computing architecture suitablefor implementing aspects of the present disclosure;

FIG. 3 illustratively depicts aspects of an example system fordetermining user activity patterns based on browser and applicationactivity across multiple user devices, in accordance with an embodimentof the present disclosure;

FIG. 4 depicts a flow diagram of a method for inferring a user activitypattern, in accordance with an embodiment of the present disclosure;

FIG. 5 depicts a flow diagram of a method for determining a probablefuture user action based on a pattern of user activity, in accordancewith an embodiment of the present disclosure;

FIG. 6 depicts a flow diagram of a method performing disambiguation todetermine a user intent, in accordance with an embodiment of the presentdisclosure;

FIG. 7 depicts a flow diagram of a method performing speech recognitionto determine an action that a user desires to be performed, inaccordance with an embodiment of the present disclosure;

FIG. 8 depict aspects the computing architecture for an automatic speechrecognition system suitable for implementing an embodiment of thepresent disclosure; and

FIG. 9 is a block diagram of an exemplary computing environment suitablefor use in implementing an embodiment of the present disclosure.

DETAILED DESCRIPTION

The subject matter of aspects of the present disclosure is describedwith specificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Aspects of the present disclosure relate to understanding user speechand inferring user intent associated with user-interactions withcomputing devices. In particular, embodiments described herein maydetermine a likely spoken utterance or intent of a user with regards toan interaction with a user device, may predict a likely future action bya user, based on a history of sensed user activity. Data correspondingto user activity may be gathered over time using sensors on one or moreuser devices associated with the user. From this historical useractivity information, a computer system may learn user activity patternsassociated with the user devices. By analyzing a user activity pattern,future user actions may be predicted or user intent may be inferred. Insome cases, the user activity pattern may be analyzed along with sensordata collected by a user device, and the user' intent inferred based ondetermining a likely intent that consistent with determined useractivity pattern.

As further described herein, in some embodiments, user devices mayemploy one or more sensors to generate data relevant to a user'sactivity via a user device(s). The user activity may be monitored,tracked, and used for determining user activity patterns. The term“activity pattern” is used broadly herein and may refer to a pluralityof user interactions conducted using one or more user devices, activityby the user on or in connection to one or more user devices, events(including actions) related to user activity, or any type of useractivity determinable via a computing device, wherein the plurality ofinteractions, actions, events, or activity share common features orcharacteristics. In some embodiments, these in-common features orvariables may comprise features characterizing a user activity, time,location or other contextual information associated with the useractivity, as further described herein. Examples of user activitypatterns may include, without limitation, activity patterns based ontime (e.g. the user browses his bank's website near the beginning ofeach month to check his account balance), location (e.g. upon arrivingat work in the morning, a user the user turns down the volume on herphone), content (e.g. a user typically browses news-related websitesfollowed by their social-media related websites), or other context, asdescribed herein.

In some embodiments, the user activity may be related to a user'sbrowsing activity, such as websites, categories or websites, orsequences of websites and/or website categories visited by a user, anduser activity associated with the browsing activity. In addition oralternatively, the user activity may be related to a user's application(or app) related activity, such as application usage, which may includeusage duration, launches, files accessed via the application or inconjunction with the application usage, or content associated with theapplication. The term application or app is used broadly herein, andgenerally refers to a computer program or computer application, whichmay comprise one or more programs or services, and may run on the user'sdevice(s) or in the cloud.

Based on the determined patterns of user activity, user intent may beinferred, regarding user interactions with a computing device, orpredictions of user activity determined and used to provide improveduser experiences. Examples of these improved user experiences, which arefurther described below, include personalization services, such astailoring content for the user, improved speech recognition or improvedsemantic understanding of the user, which may be used for performingdisambiguation or other aspects of understanding input from the user. Insome embodiments, the user activity patterns, or the inferred intent orpredictions of future activity determined therefrom, may be madeavailable to one or more applications and services that consume thisinformation and provide an improved user experience, such as avoice-controlled user interface, which may be a component of a vehicle,smart appliance, or other devices for which it is beneficial to have asemantic understanding of the user intent, or may be incorporated into alanguage model or spoken language understanding model (SLU), or similarcomputer process, component, or service. In one embodiment, the activitypattern information may be provided via an API so that third-partyapplications and services can use it, such as by providingspeech-to-text services, voice operation or control, timelyrecommendations, suggestions or other information or services relevantto the user based on the learned activity patterns. One embodiment maybe incorporated into (or operate in conjunction with) an automaticspeech recognition (ASR) and/or language modeling component or service.In this manner, operating in conjunction with (or as a component of) thespeech recognition and interpretation operation, the user's utterancemay be resolved in such a manner as to be consistent with learnedpatterns of user activity. Such embodiments can provide a significantadvantage to personal digital assistant services and an improvement tothe technology of speech recognition and understanding by enablingcomputer-recognized speech functionality to “learn” or evolve tounderstand the user.

Accordingly, at a high level, in one embodiment, user data is receivedfrom one or more data sources. The user data may be received bycollecting user data with one or more sensors or components on userdevice(s) associated with a user. Examples of user data, also describedin connection to component 210 of FIG. 2, may include information aboutthe user device(s), user-activity associated with the user devices(e.g., app usage, online activity, searches, calls, usage duration, andother user-interaction data), network-related data (such as network ID,connection data, or other network-related information), applicationdata, contacts data, calendar and social network data, or nearly anyother source of user data that may be sensed or determined by a userdevice or other computing device. The received user data may bemonitored and information about the user activity may be stored in auser profile, such as user profile 240 of FIG. 2.

In some embodiments, based on an identification of one or more userdevices, which may be determined from the user data, the one or moreuser devices are monitored for user activity. In some embodiments useractivity monitoring may be facilitated using an application or servicethat runs on the monitored user device. Alternatively or in addition,the user activity monitoring may be facilitated using an application orservice that runs in the cloud, which may scans the user device, ordetects online activity associated with the user device, such as httprequests or other communication information, or otherwise receiveinformation about user activity from the user device.

User data may be analyzed to detect various features associated withuser actions. Detected user actions or “activity events”, which mayinclude actions such as websites visited, applications launched, orother actions similar to those described herein, may be logged withassociated contextual data; for example, by logging the observed useraction with a corresponding timestamp, location stamp, and/orassociating the activity event with other available contextualinformation. In some embodiments, such logging may be performed on eachuser device, so that activity patterns may be determined across devices.Further, in some embodiments, cloud-based user-activity informationsources may be used such as online user calendars or user activitiesdetermined from social media posts, emails, or the like. These sourcesalso may be used for providing other context to the user activitydetected on the user devise(s). Examples of contextual information arefurther described in connection to contextual information extractor 284in FIG. 2. In some embodiments, user activity logs from multiple userdevices and available user activity information from cloud-based sourcesmay be combined thereby representing a composite user activity history.The user activity logs, including corresponding contextual information,may be stored in a user profile associated with the user, such as userprofile 240 of FIG. 2.

From the activity logs or user activity data, historical user activityinformation may be determined and provided to an inference engine. Basedon an analysis of historical user activity, and in some cases currentsensor data regarding user activity, a set of one or more likely useractivity patterns may be determined. In particular, the inference enginemay conduct an analysis of the historical user activity information toidentify user activity patterns, which may be determined by detectingrepetitions of similar user actions or routines, in some embodiments.

In some embodiments, a corresponding confidence weight or confidencescore may be determined regarding the user activity patterns. Theconfidence score may be based on the strength of the pattern, which maybe determined by the number of observations used to determine a pattern,how frequently the user activity is consistent with the pattern, the ageor freshness of the activity observations, the number of features incommon with the activity observations that make up the pattern, orsimilar measurements. In some instances, the confidence score may beconsidered when providing a personalized user experience or otherimproved user experience. Further, in some embodiments, a minimumconfidence score may be needed before using the activity pattern toprovide such experiences or other services. For example, in oneembodiment, a threshold of 0.6 (or just over fifty percent) is utilizedsuch that only activity patterns having a 0.6 (or greater) likelihood ofpredicting user activity may be considered. Nevertheless, whereconfidence scores and thresholds are used, determined patterns of useractivity with confidence scores less than the threshold may still bemonitored since additional observations of user activities may increasethe confidence for a particular pattern.

In some embodiments, crowdsourced user activity history may also beutilized in conjunction with a user's own activity history. For example,for a given user, a set of other users similar to the given user may beidentified, based on having features or characteristics in common withthe given user. This might include other users located in proximity tothe given users, the given user's social media friends, work colleagues(which may be determined from an analysis of contextual informationassociated with the given user), other users with similar users activitypatterns, or the like. Information about user activity history from theother users may be relied upon for inferring patterns of user activityfor the given user. This may be particularly useful in situations wherelittle user activity history exists for the given user, such as wherethe user is a new user. In some embodiments, user activity informationfrom similar users may be imputed to the new user until enough userhistory is available for the new user to determine statisticallyreliable user pattern predictions, which may be determined based on thenumber of observations included in the user activity history informationor the statistical confidence of the determined user activity patterns,as further described herein. In some cases, where the user activityhistory comes from other users, the resulting inferred activity patternsfor the given user may be assigned a lower confidence.

In some embodiments, a semantic analysis is performed on the activityevents and associated information to characterize aspects of theactivity event. For example, features associated with an activity eventmay be categorized (such as by type, similar timeframe or location, forexample), and related features may be identified for use in determininga similarity or relational proximity to other activity events, such asby having one or more of the characteristics, including categories andrelated features, in common. In some embodiments, a semantic knowledgerepresentation, such as a relational knowledge graph, may be employed.In some embodiments, the semantic analysis may use rules, logic such asassociations or conditions, or classifiers.

The semantic analysis may also be used to characterize informationassociated with the activity event, such as determining that a locationassociated with the activity corresponds to a hub or venue of interestto the user based on frequency of visits, such as the user's home, work,gym, etc. (For example, the user's home hub may be determined to be thelocation where the user spends most of her time between 8 PM and 6 AM.)Similarly, the semantic analysis may determine time of day thatcorrespond to working hours, lunch time, commute time, etc.

In this way, the semantic analysis may provide other relevant featuresof the activity events that may be used to determine patterns. Forexample, in addition to determining a particular website that the uservisited at a certain time, such as visiting CNN.com over lunch, thecategory of the website may be determined, such as a news-relatedwebsite. Similarly, the semantic analysis may categorize the activity asbeing associated with work or home, based on the characteristics of theactivity (e.g. a batch of online searches about chi-squared distributionthat occurs during working hours at a location corresponding to theuser's office may be determined to be work-related activity, whereasstreaming a movie on Friday night at a location corresponding to theuser's home may be determined to be home-related activity). Theseaspects characterizing the user activity event then may be consideredwhen evaluating the user activity history to identify patterns. Forexample, a pattern of visiting news-related websites over lunch may bedetermined where a user routinely visits news-related websites overlunch, but only occasionally visits CNN.com as one of the news-relatedwebsites.

As described previously, the user activity patterns may be used to inferuser intent or predict future activity of the user. From thesepredictions or inferred user intent, various implementations may provideimproved user experiences. For example, some embodiments may providetimely, relevant delivery or presentation of content, or modifyingcontent that would otherwise be presented; incorporation of the user'sroutine into recommendations and notifications; improved speechrecognition and improved semantic understanding by the user's computerdevices, or other examples of improved user experiences describedherein. For example, acoustic information corresponding to a user'sspeech, such as a spoken command, query, question, or other data fromother user interactions with a user device (which may include visualinformation from a gesture that may be sensed by a sensor, such as acamera, associated with the user device) may be received and interpretedbased on the historical user activity patterns. The information may beinterpreted, for instance, to enable a personal assistant application torespond or invoke some action on behalf of the user (such as set areminder, navigate to a website, locate a document for the user, orother tasks that may be carried out via the user device. Someembodiments may be carried out by a personal assistant application orservice, which may be implemented as one or more computer applications,services, or routines, such as an app running on a mobile device or thecloud, as further described herein.

Turning now to FIG. 1, a block diagram is provided showing an exampleoperating environment 100 in which some embodiments of the presentdisclosure may be employed. It should be understood that this and otherarrangements described herein are set forth only as examples. Otherarrangements and elements (e.g., machines, interfaces, functions,orders, and groupings of functions, etc.) can be used in addition to orinstead of those shown, and some elements may be omitted altogether forthe sake of clarity. Further, many of the elements described herein arefunctional entities that may be implemented as discrete or distributedcomponents or in conjunction with other components, and in any suitablecombination and location. Various functions described herein as beingperformed by one or more entities may be carried out by hardware,firmware, and/or software. For instance, some functions may be carriedout by a processor executing instructions stored in memory.

Among other components not shown, example operating environment 100includes a number of user devices, such as user devices 102 a and 102 bthrough 102 n; a number of data sources, such as data sources 104 a and104 b through 104 n; server 106; sensors 103 a and 107, and network 110.It should be understood that environment 100 shown in FIG. 1 is anexample of one suitable operating environment. Each of the componentsshown in FIG. 1 may be implemented via any type of computing device,such as computing device 900 described in connection to FIG. 9, forexample. These components may communicate with each other via network110, which may include, without limitation, one or more local areanetworks (LANs) and/or wide area networks (WANs). In exemplaryimplementations, network 110 comprises the Internet and/or a cellularnetwork, amongst any of a variety of possible public and/or privatenetworks.

It should be understood that any number of user devices, servers, anddata sources may be employed within operating environment 100 within thescope of the present disclosure. Each may comprise a single device ormultiple devices cooperating in a distributed environment. For instance,server 106 may be provided via multiple devices arranged in adistributed environment that collectively provide the functionalitydescribed herein. Additionally, other components not shown may also beincluded within the distributed environment.

User devices 102 a and 102 b through 102 n may be client devices on theclient-side of operating environment 100, while server 106 may be on theserver-side of operating environment 100. Server 106 can compriseserver-side software designed to work in conjunction with client-sidesoftware on user devices 102 a and 102 b through 102 n so as toimplement any combination of the features and functionalities discussedin the present disclosure. This division of operating environment 100 isprovided to illustrate one example of a suitable environment, and thereis no requirement for each implementation that any combination of server106 and user devices 102 a and 102 b through 102 n remain as separateentities.

User devices 102 a and 102 b through 102 n may comprise any type ofcomputing device capable of use by a user. For example, in oneembodiment, user devices 102 a through 102 n may be the type ofcomputing device described in relation to FIG. 9 herein. By way ofexample and not limitation, a user device may be embodied as a personalcomputer (PC), a laptop computer, a mobile or mobile device, asmartphone, a tablet computer, a smart watch, a wearable computer, apersonal digital assistant (PDA), an MP3 player, a global positioningsystem (GPS) or device, a video player, a handheld communicationsdevice, a gaming device or system, an entertainment system, a vehiclecomputer system, an embedded system controller, a camera, a remotecontrol, a bar code scanner, a computerized measuring device, anappliance, a consumer electronic device, a workstation, or anycombination of these delineated devices, or any other suitable device.

Data sources 104 a and 104 b through 104 n may comprise data sourcesand/or data systems, which are configured to make data available to anyof the various constituents of operating environment 100, or system 200described in connection to FIG. 2. (For example, in one embodiment, oneor more data sources 104 a through 104 n provide (or make available foraccessing) user data to user-data collection component 210 of FIG. 2.)Data sources 104 a and 104 b through 104 n may be discrete from userdevices 102 a and 102 b through 102 n and server 106 or may beincorporated and/or integrated into at least one of those components. Inone embodiment, one or more of data sources 104 a though 104 n compriseone or more sensors, which may be integrated into or associated with oneor more of the user device(s) 102 a, 102 b, or 102 n or server 106.Examples of sensed user data made available by data sources 104 a though104 n are described further in connection to user-data collectioncomponent 210 of FIG. 2.

Operating environment 100 can be utilized to implement one or more ofthe components of system 200, described in FIG. 2, and system 300,described in FIG. 3, including components for collecting user data,monitoring activity events, determining activity patterns, consumingactivity pattern information to provide an improving user experience,generating personalized content, and/or presenting notifications andrelated content to users. Referring now to FIG. 2, with FIG. 1, a blockdiagram is provided showing aspects of an example computing systemarchitecture suitable for implementing an embodiment and designatedgenerally as system 200. System 200 represents only one example of asuitable computing system architecture. Other arrangements and elementscan be used in addition to or instead of those shown, and some elementsmay be omitted altogether for the sake of clarity. Further, as withoperating environment 100, many of the elements described herein arefunctional entities that may be implemented as discrete or distributedcomponents or in conjunction with other components, and in any suitablecombination and location.

Example system 200 includes network 110, which is described inconnection to FIG. 1, and which communicatively couples components ofsystem 200 including user-data collection component 210, presentationcomponent 220, user activity monitor 280, activity pattern inferenceengine 260, activity pattern consumers 270, and storage 225. Useractivity monitor 280 (including its components 282, 284, and 286),activity pattern inference engine 260 (including its components 262,264, 266, 267, and 269), user-data collection component 210,presentation component 210, and activity pattern consumers 270 may beembodied as a set of compiled computer instructions or functions,program modules, computer software services, or an arrangement ofprocesses carried out on one or more computer systems, such as computingdevice 900 described in connection to FIG. 9, for example.

In one embodiment, the functions performed by components of system 200are associated with one or more personal assistant applications,services, or routines. In particular, such applications, services, orroutines may operate on one or more user devices (such as user device102 a), servers (such as server 106), may be distributed across one ormore user devices and servers, or be implemented in the cloud. Moreover,in some embodiments, these components of system 200 may be distributedacross a network, including one or more servers (such as server 106) andclient devices (such as user device 102 a), in the cloud, or may resideon a user device, such as user device 102 a. Moreover, these components,functions performed by these components, or services carried out bythese components may be implemented at appropriate abstraction layer(s)such as the operating system layer, application layer, hardware layer,etc., of the computing system(s). Alternatively, or in addition, thefunctionality of these components and/or the embodiments describedherein can be performed, at least in part, by one or more hardware logiccomponents. For example, and without limitation, illustrative types ofhardware logic components that can be used include Field-programmableGate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs),Application-specific Standard Products (ASSPs), System-on-a-chip systems(SOCs), Complex Programmable Logic Devices (CPLDs), etc. Additionally,although functionality is described herein with regards to specificcomponents shown in example system 200, it is contemplated that in someembodiments functionality of these components can be shared ordistributed across other components.

Continuing with FIG. 2, user-data collection component 210 is generallyresponsible for accessing or receiving (and in some cases alsoidentifying) user data from one or more data sources, such as datasources 104 a and 104 b through 104 n of FIG. 1. In some embodiments,user-data collection component 210 may be employed to facilitate theaccumulation of user data of a particular user (or in some cases, aplurality of users including crowd-sourced data) for user activitymonitor 280, activity pattern inference engine 260, or an activitypattern consumer 270. The data may be received (or accessed), andoptionally accumulated, reformatted and/or combined, by data collectioncomponent 210 and stored in one or more data stores such as storage 225,where it may be available to other components of system 200. Forexample, the user data may be stored in or associated with a userprofile 240, as described herein. In some embodiments, any personallyidentifying data (i.e. user data that specifically identifies particularusers) is either not uploaded or otherwise provided from the one or moredata sources with user data, is not permanently stored, and/or is notmade available to user activity monitor 280 and/o activity patterninference engine 260.

User data may be received from a variety of sources where the data maybe available in a variety of formats. For example, in some embodiments,user data received via user-data collection component 210 may bedetermined via one or more sensors (such as sensors 103 a and 107 ofFIG. 1), which may be on or associated with one or more user devices(such as user device 102 a), servers (such as server 106), and/or othercomputing devices. As used herein, a sensor may include a function,routine, component, or combination thereof for sensing, detecting, orotherwise obtaining information such as user data from a data source 104a, and may be embodied as hardware, software, or both. By way of exampleand not limitation, user data may include data that is sensed ordetermined from one or more sensors (referred to herein as sensor data),such as location information of mobile device(s), properties orcharacteristics of the user device(s) (such as device state, chargingdata, date/time, or other information derived from a user device such asa mobile device), user-activity information (for example: app usage;online activity; searches; voice data such as automatic speechrecognition; activity logs; communications data including calls, texts,instant messages, and emails; website posts; other user-data associatedwith communication events; etc.) including, in some embodiments, useractivity that occurs over more than one user device, user history,session logs, application data, contacts data, calendar and scheduledata, notification data, social-network data, news (including popular ortrending items on search engines or social networks), online gamingdata, ecommerce activity (including data from online accounts such asMicrosoft®, Amazon.com®, Google®, eBay®, PayPal®, video-streamingservices, gaming services, or Xbox Live®), user-account(s) data (whichmay include data from user preferences or settings associated withassociated with a personalization-related application, a personalassistant application or service), home-sensor data, appliance data,global positioning system (GPS) data, vehicle signal data, traffic data,weather data (including forecasts), wearable device data, other userdevice data (which may include device settings, profiles,network-related information (e.g., network name or ID, domaininformation, workgroup information, connection data, Wi-Fi network data,or configuration data, data regarding the model number, firmware, orequipment, device pairings, such as where a user has a mobile phonepaired with a Bluetooth headset, for example, or other network-relatedinformation), gyroscope data, accelerometer data, payment or credit cardusage data (which may include information from a user's PayPal account),purchase history data (such as information from a user's Xbox Live,Amazon.com, or eBay account), other sensor data that may be sensed orotherwise detected by a sensor (or other detector) component(s)including data derived from a sensor component associated with the user(including location, motion, orientation, position, user-access,user-activity, network-access, user-device-charging, or other data thatis capable of being provided by one or more sensor component), dataderived based on other data (for example, location data that can bederived from Wi-Fi, Cellular network, or IP address data), and nearlyany other source of data that may be sensed or determined as describedherein.

In some respects, user data may be provided in user-data streams orsignals. A “user signal” can be a feed or stream of user data from acorresponding data source. For example, a user signal could be from asmartphone, a home-sensor device, a GPS device (e.g., for locationcoordinates), a vehicle-sensor device, a wearable device, a user device,a gyroscope sensor, an accelerometer sensor, a calendar service, anemail account, a credit card account, or other data sources. In someembodiments, user-data collection component 210 receives or accessesdata continuously, periodically, or as needed.

User activity monitor 280 is generally responsible for monitoring userdata for information that may be used for determining user activityinformation, which may include identifying and/or tracking features(sometimes referred to herein as “variables”) or other informationregarding specific user actions and related contextual information.Embodiments of user activity monitor 280 may determine, from themonitored user data, user activity associated with a particular user. Asdescribed previously, the user activity information determined by useractivity monitor 280 may include user activity information from multipleuser devices associated with the user and/or from cloud-based servicesassociated with the user (such as email, calendars, social-media, orsimilar information sources), and which may include contextualinformation associated with the identified user activity. User activitymonitor 280 may determine current or near-real-time user activityinformation and may also determine historical user activity information,in some embodiments, which may be determined based on gatheringobservations of user activity over time, accessing user logs of pastactivity (such as browsing history, for example). Further, in someembodiments, user activity monitor 280 may determine user activity(which may include historical activity) from other similar users (i.e.crowdsourcing), as described previously.

In some embodiments, information determined by user activity monitor 280may be provided to activity pattern inference engine 260 includinginformation regarding the current context and historical visits(historical observations). Some embodiments may also provide useractivity information, such as current user activity, to one or moreactivity pattern consumers 270. As described previously, user activityfeatures may be determined by monitoring user data received fromuser-data collection component 210. In some embodiments, the user dataand/or information about the user activity determined from the user datais stored in a user profile, such as user profile 240.

In an embodiment, user activity monitor 280 comprises one or moreapplications or services that analyze information detected via one ormore user devices used by the user and/or cloud-based servicesassociated with the user, to determine activity information and relatedcontextual information. Information about user devices associated with auser may be determined from the user data made available via user-datacollection component 210, and maybe provided to user activity monitor280, activity pattern inference engine 270, or other components ofsystem 200.

More specifically, in some implementations of user activity monitor 280,a user device may be identified by detecting and analyzingcharacteristics of the user device, such as device hardware, softwaresuch as operating system (OS), network-related characteristics, useraccounts accessed via the device, and similar characteristics. Forexample, information about a user device may be determined usingfunctionality of many operating systems to provide information about thehardware, OS version, network connection information, installedapplication, or the like.

Some embodiments of user activity monitor 280, or its subcomponents, maydetermine a device name or identification (device ID) for each deviceassociated with a user. This information about the identified userdevices associated with a user may be stored in a user profileassociated with the user, such as in user accounts and devices 244 ofuser profile 240. In an embodiment, the user devices may be polled,interrogated, or otherwise analyzed to determine information about thedevices. This information may be used for determining a label oridentification of the device (e.g. a device id) so that the userinteraction with device may be recognized from user data by useractivity monitor 280. In some embodiments, users may declare or registera device, such as by logging into an account via the device, installingan application on the device, connecting to an online service thatinterrogates the device, or otherwise providing information about thedevice to an application or service. In some embodiments devices thatsign into an account associated with the user, such as a Microsoft®account or Net Passport, email account, social network, or the like, areidentified and determined to be associated with the user.

As shown in example system 200, user activity monitor 280 comprises auser activity detector 282, contextual information extractor 284, and anactivity features determiner 286. In some embodiments, user activitymonitor 280, one or more of its subcomponents, or other components ofsystem 200, such as activity pattern consumers 270 or activity patterninference engine 260, may determine interpretive data from received userdata. Interpretive data corresponds to data utilized by these componentsof system 200 or subcomponents of user activity monitor 280 to interpretuser data. For example, interpretive data can be used to provide othercontext to user data, which can support determinations or inferencesmade by the components or subcomponents. Moreover, it is contemplatedthat embodiments of user activity monitor 280, its subcomponents, andother components of system 200 may use user data and/or user data incombination with interpretive data for carrying out the objectives ofthe subcomponents described herein. Additionally, although severalexamples of how user activity monitor 280 and its subcomponents mayidentify user activity information are described herein, many variationsof user activity identification and user activity monitoring arepossible in various embodiments of the disclosure.

User activity detector 282, in general, is responsible for determining(or identifying) a user action or activity event has occurred.Embodiments of activity detector 282 may be used for determining currentuser activity or one or more historical user actions. Some embodimentsof activity detector 282 may monitor user data for activity-relatedfeatures or variables corresponding to user activity such as indicationsof applications launched or accessed, files accessed, modified, copied,etc., websites navigated to, online content downloaded and rendered orplayed, or similar user activities.

Additionally, some embodiments of user activity detector 282 extractfrom the user data information about user activity, which may includecurrent user activity, historical user activity, and/or relatedinformation such as contextual information. (Alternatively or inaddition, in some embodiments contextual information extractor 284determines and extracts contextual information. Similarly, in someembodiments, activity features determiner 286 extract information aboutuser activity, such user activity related features, based on anidentification of the activity determined by user activity detector282.) Examples of extracted user activity information may include appusage, online activity, searches, calls, usage duration, applicationdata (e.g. emails, messages, posts, user status, notifications, etc.),or nearly any other data related to user interactions with the userdevice or user activity via a user device. Among other components ofsystem 200, the extracted user activity information determined byactivity detector 282 may be provided to other subcomponents of useractivity monitor 280, activity pattern inference engine 260, or one ormore activity pattern consumers 270. Further, the extracted useractivity may be stored in a user profile associated with the user, suchas in user activity information component 242 of user profile 240. Insome embodiments, activity detector 282 or user activity monitor 280 (orits other sub components) performs conflation on the detected useractivity information. For example, overlapping information may be mergedand duplicated or redundant information eliminated.

In some embodiments, the user activity-related features may beinterpreted to determine a user activity has occurred. For example, insome embodiments, activity detector 282 employs user activity eventlogic, which may include rules, conditions, associations, classificationmodels, or other criteria to identify user activity. For example, in oneembodiment, user activity event logic may include comparing useractivity criteria with the user data in order to determine that anactivity event has occurred. The activity event logic can take manydifferent forms depending on the mechanism used to identify an activityevent. For example, the user activity event logic could be training dataused to train a neural network that is used to evaluate user data todetermine when an activity event has occurred. The activity event logicmay comprise fuzzy logic, neural network, finite state machine, supportvector machine, logistic regression, clustering, or machine learningtechniques, similar statistical classification processes or,combinations of these to identify activity events from user data. Forexample, activity event logic may specify types of user deviceinteraction(s) information that are associated with an activity event,such as a navigating to a website, composing an email, or launching anapp. In some embodiments, a series or sequence of user deviceinteractions may be mapped to an activity event, such that the activityevent may be detected upon determining that the user data indicates theseries or sequence of user interactions has been carried out by theuser.

In some embodiments, activity event logic maybe specify types ofuser-device related activity that is considered user activity, such asactivity that happens while a user is logged into the user device, whileuser interfaces are receiving input (e.g. while a computer mouse,touchpad, screen, voice-recognition interface, or the like are active),or certain types of activity like launching applications, modifyingfiles with applications, opening a browser and navigating to a website,etc. In this way, the activity event logic may be used to distinguishgenuine user activity from automated activity of processes running onthe user devices, such as automatic updates or malware scanning. Once auser activity is determined, these features or additional relatedfeatures may be detected and associated with the detected activity foruse in determining activity patterns.

In some embodiments, user activity detector 282 runs on or inassociation with each user device for a user. Activity detector 282 mayinclude functionality that polls or analyzes aspects of the operatingsystem to determine user activity related features (such as installed orrunning applications or file accesses and modifications, for example)network communications, and/or other user actions detectable via theuser device including sequences of actions.

In some embodiments, such as the embodiment shown in system 200, useractivity detector 282 includes subcomponents comprising an app activitylogging pipeline 283 and a browse activity logging pipeline 285. Theselogging pipelines may be embodied as client-side applications orservices that run on each user device associated with a user, and insome embodiments may run in conjunction with applications or inside (oras a part of) applications, such as within a browser or as a browserplug-in or extension. App activity logging pipeline 283, in general,manages logging of a user's application (or app) activity, such asapplication download, launch, access, use (which may include duration),file access via the application, and in-application user activity (whichmay include application content). Browse activity logging pipeline 285,in general, manages logging of a user's browse activity, such aswebsites visited, social media activity (which may include browse-typeactivity conducted via specific browsers or apps like the Facebook® app,Twitter® app, Instagram® app, Pinterest® app, etc.) content downloaded,files accessed, and other browse-related user activity. In someembodiments, each browser on a user device is associated with aninstance of browse activity logging pipeline 285, or alternatively aplugin or service that provided browse information to a single instanceof browse activity logging pipeline 285 on the user device. In someembodiments, app and browse activity logging pipelines 283 and 285 mayalso perform functionality described in connection with contextualinformation extractor 284, such as logging timestamps, location stamps,user-device related information, or other contextual information that isassociated with the logged app activity or browse activity. In someembodiments, app and browse activity logging pipelines 283 and 285upload logged user activity information to activity pattern inferenceengine 260 and/or store the logged activity information in a userprofile associated with the user, such as in user activity informationcomponent 242 of user profile 240.

Contextual information extractor 284, in general, is responsible fordetermining contextual information related to the user activity(detected by user activity detector 282 or user activity monitor 280),such as context features or variables associated with user activity,related information, and user-related activity, and further responsiblefor associating the determined contextual information with the detecteduser activity. In some embodiments, contextual information extractor 284may associate the determined contextual information with the relateduser activity and may also log the contextual information with theassociated user activity. Alternatively, the association or logging maybe carried out by another service. For example, some embodiments ofcontextual information extractor 284 provide the determined contextualinformation to activity features determiner 286, which determinesactivity features of the user activity and/or related contextualinformation.

Some embodiments of contextual information extractor 284 determinecontextual information related to a user action or activity event suchas entities identified in a user activity or related to the activity(e.g., recipients of a group email sent by the user), which may includenicknames used by the user (e.g., “mom” and “dad” referring to specificentities who may be identified in the user's contacts by their actualnames), user activity associated with the location or venue of theuser's device. By way of example and not limitation, this may includecontext features such as location data; which may be represented as alocation stamp associated with the activity; contextual informationabout the location, such as venue information (e.g. this is the user'soffice location, home location, school, restaurant, move theater, etc.),yellow pages identifier (YPID) information, time, day, and/or date,which may be represented as a timestamp associated with the activity;user device characteristics or user device identification informationregarding the device on which the user carried out the activity;duration of the user activity, other user activity/activities precedingand/or following the user activity (such as sequences of websitesvisited, a sequence of online searches conducted, sequences ofapplication and website usage, such as browsing to a bank and thenaccessing an Excel® spreadsheet file to record financial information, orthe like), other information about the activity such as entitiesassociated with the activity (e.g. venues, people, objects, etc.) whichmay include nicknames or personal expressions or terms used by (and insome instances created by) the user or acquaintances of the user (forexample, a name for the venue that is specific to the user but noteveryone, such as “Dikla's home,” “Shira's car,” “my Seattle friends,”etc.), information detected by sensor(s) on user devices associated withthe user that is concurrent or substantially concurrent to the useractivity (e.g. motion information or physiological information detectedon a fitness tracking user device, listening to music, which maydetected via a microphone sensor if the source of the music is not auser device), or any other information related to the user activity thatis detectable that may be used for determining patterns of useractivity.

In embodiments using contextual information related to user devices, auser device may be identified by detecting and analyzing characteristicsof the user device, such as device hardware, software such as operatingsystem (OS), network-related characteristics, user accounts accessed viathe device, and similar characteristics. For example, as describedpreviously, information about a user device may be determined usingfunctionality of many operating systems to provide information about thehardware, OS version, network connection information, installedapplication, or the like. In some embodiments, a device name oridentification (device ID) may be determined for each device associatedwith a user. This information about the identified user devicesassociated with a user may be stored in a user profile associated withthe user, such as in user account(s) and device(s) 244 of user profile240. In an embodiment, the user devices may be polled, interrogated, orotherwise analyzed to determine contextual information about thedevices. This information may be used for determining a label oridentification of the device (e.g. a device id) so that user activity onone user device may be recognized and distinguished from user activityon another user device. Further, as described previously, in someembodiments, users may declare or register a user device, such as bylogging into an account via the device, installing an application on thedevice, connecting to an online service that interrogates the device, orotherwise providing information about the device to an application orservice. In some embodiments devices that sign into an accountassociated with the user, such as a Microsoft® account or Net Passport,email account, social network, or the like, are identified anddetermined to be associated with the user.

In some implementations, contextual information extractor 284 mayreceive user data from user-data collection component 210, parse thedata, in some instances, and identify and extract context features orvariables (which may also be carried out by activity features determiner286). Context variables may be stored as a related set of contextualinformation associated with the user activity, and may be stored in auser profile such as in user activity information component 242. In somecases, contextual information may be used by one or more activitypattern consumers, such as for personalizing content or a userexperience, such as when, where, or how to present content. Contextualinformation also may be determined from the user data of one or moreusers, in some embodiments, which may be provided by user-datacollection component 210 in lieu of or in addition to user activityinformation for the particular user.

Activity features determiner 286 is generally responsible fordetermining activity-related features (or variables) associated with theuser activity that may be used for identifying patterns of useractivity. Activity features may be determined from information about auser activity and/or from related contextual information. In someembodiments, activity features determiner 286 receives user-activity orrelated information from user activity monitor 280 (or itssubcomponents), and analyzes the received information to determine a setof one or more features associated with the user activity.

Examples of activity-related features include, without limitation,location-related features, such as location of the user device(s) duringthe user activity, venue-related information associated with thelocation, or other location-related information; time related features,such as time(s) of day(s), day of week or month the user activity, orthe duration of the activity, or related duration information such ashow long the user used an application associated with the activity; userdevice-related features, such as device type (e.g. desktop, tablet,mobile phone, fitness tracker, heart rate monitor, etc.) hardwareproperties or profiles, OS or firmware properties, device IDs or modelnumbers, network-related information (e.g. mac address, network name, IPaddress, domain, work group, information about other devices detected onthe local network, router information, proxy or VPN information, othernetwork connection information, etc.), position/motion/orientationrelated information about the user device, power information such asbattery level, time of connecting/disconnecting a charger,user-access/touch information; usage related features, such as file(s)accessed, app usage (which may also include application data, in-appusage, concurrently running applications), network usage information,user account(s) accessed or otherwise used, (such as device account(s),OS level account(s), or online/cloud-services related account(s)activity, such as Microsoft® account or Net Passport, online storageaccount(s), email, calendar, or social networking accounts, etc.;content-related features, such as online activity (e.g. searches,browsed websites, purchases, social networking activity, communicationssent or received including social media posts; other features that maybe detected concurrent with the user activity or near the time or theuser activity; or any other features that may be detected or sensed andused for determining a pattern of the user activity. Features may alsoinclude information about user(s) using the device; other informationidentifying a user, such as a login password, biometric data, which maybe provided by a fitness tracker or biometric scanner; and/orcharacteristics of the user(s) who use the device, which may be usefulfor distinguishing users on devices that are shared by more than oneuser. In some embodiments, user activity event logic (described inconnection to user activity detector 282) may be utilized to identifyspecific features from user activity information.

Continuing with system 200 of FIG. 2, activity pattern inference engine260 is generally responsible for determining user activity patternsbased on the user activity information determined from user activitymonitor 280. In some embodiments, activity pattern inference engine 260may run on a server, as a distributed application across multipledevices, or in the cloud. At a high level, activity pattern inferenceengine 260 may receive user-activity-related information, which may beuploaded from user-activity logs from client-side applications orservices associated with user activity monitor 280. One or moreinference algorithms may be applied to the user-activity-relatedinformation to determine a set of likely user activity patterns. Forexample, patterns may be determined based on similar instances ofobservation of user activity or associated contextual information, whichmay be referred to as “in-common features” of user activity-relatedinformation. The inferred activity pattern information may be providedto an activity pattern consumer 270 and/or used to generate a patternbased prediction regarding likely future user action(s). In someembodiments, a corresponding confidence is also determined for thepatterns (or predictions based on the patterns), as described herein.Further, the activity pattern (or prediction of future action based on apattern) may comprise a single (future-occurring) user activity likelyto occur, a sequence of future user actions, or probabilities for morethan one future action; for example, and eighty percent likelihood thatthe next action will be browsing to website A and a fifteen percentlikelihood that the next action will be launching a music playerapplication, and a five percent likelihood that the next action will bebrowsing to website B.

As shown in example system 200, activity pattern inference engine 260comprises semantic information analyzer 262, features similarityidentifier 264, and activity pattern determiner 266. Semanticinformation analyzer 262 is generally responsible for determiningsemantic information associated with the user-activity related featuresidentified by user activity monitor 280. For example, while auser-activity feature may indicate a specific website visited by theuser, semantic analysis may determine the category of website, relatedwebsites, themes or topics or other entities associated with the websiteor user activity. Semantic information analyzer 262 may determineadditional user-activity related features semantically related to theuser activity, which may be used for identifying user activity patterns.

In particular, as described previously, a semantic analysis is performedon the user activity information, which may include the contextualinformation, to characterize aspects of the user action or activityevent. For example, in some embodiments, activity features associatedwith an activity event may be classified or categorized (such as bytype, timeframe or location, work-related, home-related, themes, relatedentities, other user(s) (such as communication to or from another user)and/or relation of the other user to the user (e.g. family member, closefriend, work acquaintance, boss, or the like), or other categories), orrelated features may be identified for use in determining a similarityor relational proximity to other user activity events, which mayindicate a pattern. In some embodiments, semantic information analyzer262 may utilize a semantic knowledge representation, such as arelational knowledge graph. Semantic information analyzer 262 may alsoutilize semantic analysis logic, including rules, conditions, orassociations to determine semantic information related to the useractivity. For example, a user activity event comprising an email sent tosomeone who works with the user may be characterized as a work-relatedactivity. Thus where the user emails some person she works with everySunday night, but not necessarily the same person, a pattern may bedetermined (using activity pattern determiner 266) that the userperforms work-related activities every Sunday night. Accordingly, it maybe appropriate to surface a notification, such as a reminder, relatingto the user's work to the user on Sunday night since the user has apattern of working on Sunday night. (Here, a notification service is oneexample of an activity pattern consumer 270).

Semantic information analyzer 262 may also be used to characterizecontextual information associated with the user activity event, such asdetermining that a location associated with the activity corresponds toa hub or venue of interest to the user (such as the user's home, work,gym, or the like) based on frequency of user visits. For example, theuser's home hub may be determined (using semantic analysis logic) to bethe location where the user spends most of her time between 8 PM and 6AM.) Similarly, the semantic analysis may determine time of day thatcorrespond to working hours, lunch time, commute time, etc. Similarly,the semantic analysis may categorize the activity as being associatedwith work or home, based on other characteristics of the activity (e.g.a batch of online searches about chi-squared distribution that occursduring working hours at a location corresponding to the user's officemay be determined to be work-related activity, whereas streaming a movieon Friday night at a location corresponding to the user's home may bedetermined to be home-related activity). In this way, the semanticanalysis provided by semantic information analyzer 262 may provide otherrelevant features of the user activity events that may be used fordetermining user activity patterns. For example, where the user activitycomprises visiting CNN.com over lunch, and the semantic analysisdetermines that the user visited a news-related website over lunch, apattern of user activity may be determined (by activity patterndeterminer 266) indicating that the user routinely visits news-relatedwebsites over lunch, but only occasionally visits CNN.com as one ofthose news-related websites.

Features similarity identifier 264 is generally responsible fordetermining similarity of activity features of two or more user activityevents (put another way, activity features characterizing a first useractivity event that are similar to activity features characterizing asecond user activity event). The activity features may include featuresrelating to contextual information and features determined by semanticanalyzer 262. Activity events having in-common activity features may beused to identify an activity pattern, which may be determined usingactivity pattern determiner 266.

For example, in some embodiments, features similarity identifier 264 maybe used in conjunction with one or more pattern based predictors 267 (asubcomponent of activity pattern determiner 266) to determine a set ofuser activity events that have in-common features. In some embodiments,this set of user activity events may be used as inputs to a patternbased predictor, as described below. In some embodiments, featuressimilarity identifier 264 comprises functionality for determiningsimilarity of periodic- and behavioral-based activity features. Periodicfeatures comprise, for example, features that may occur periodically;for example, features on a day of the week or month, even/odd days (orweeks), monthly, yearly, every other day, every 3^(rd) day, etc.Behavior features may comprise behaviors such as user activities thattend to occur with certain locations or activities occurring before orafter a given user activity event (or sequence of previous activityevents), for example.

In embodiments where activity features have a value, similarity may bedetermined among different activity features having the same value orapproximately the same value, based on the particular feature. (Forexample, a timestamp of a first activity happening at 9:01 AM on Fridayand a timestamp of a second activity happening at 9:07 AM on Friday maybe determined to have similar or in-common timestamp features.)

Activity pattern determiner 266 is generally responsible for determininga user activity pattern based on similarities identified in useractivity information. In particular, activity pattern determiner 266 (oractivity pattern inference engine 260) may determine a user activitypattern based on repetitions of similar activity features associatedwith a plurality of observed activity events. Thus for example, anactivity pattern may be determined where activity features correspondingto two or more activity events are similar. In some instances, anactivity event may have many corresponding activity features, which maybe represented as a feature vector associated with a particular activityevent. Accordingly, the analysis carried out by activity patterndeterminer 266 may involve comparing the activity features from featuresvectors of plurality of activity events.

In some embodiments, activity patterns may be determined using patterninferences logic 230. Pattern inferences logic may include rules,associations, conditions, prediction and/or classification models, orpattern inference algorithms The pattern inferences logic 230 can takemany different forms depending on the particular activity pattern or themechanism used to identify an activity pattern, or identify featuresimilarity among observed activity events to determine the pattern. Forexample, some embodiments of pattern inferences logic 230 may employmachine learning mechanisms to determine feature similarity, or otherstatistical measures to determine the activity events belonging to a setof “example user actions” that support the determined activity pattern,as further described below. The user activity information may bereceived from user activity monitor 280 and information about identifiedsimilar features may be received from features similarity identifier264. In some embodiments, the user pattern(s) determined by activitypattern determiner 266 may be stored as inferred user routines 248 inuser profile 240.

In some embodiments, activity pattern determiner 266 provides a patternof user activity and an associated confidence score regarding thestrength of the user pattern, which may reflect the likelihood thatfuture user activity will follow the pattern. More specifically, in someembodiments, a corresponding confidence weight or confidence score maybe determined regarding a determined user activity pattern. Theconfidence score may be based on the strength of the pattern, which maybe determined based on the number of observations (of a particular useractivity event) used to determine a pattern, how frequently the user'sactions are consistent with the activity pattern, the age or freshnessof the activity observations, the number of similar features, types offeatures, and/or degree of similarity of the features in common with theactivity observations that make up the pattern, or similar measurements.

In some instances, the confidence score may be considered when providinga determined activity pattern to an activity pattern consumer 270. Forexample, in some embodiments, a minimum confidence score may be neededbefore using the activity pattern to provide an improved user experienceor other service by an activity pattern consumer 270. In one embodiment,a threshold of 0.6 (or just over fifty percent) is utilized such thatonly activity patterns having a 0.6 (or greater) likelihood ofpredicting user activity may be may be provided. Nevertheless, whereconfidence scores and thresholds are used, determined patterns of useractivity with confidence scores less than the threshold still may bemonitored and updated based on additional activity observations, sincethe additional observations of may increase the confidence for aparticular pattern.

Some embodiments of activity pattern determiner 266 determine a patternaccording to the example approaches described below, where each instanceof a user activity event has corresponding historical values of trackedactivity features (variables) that form patterns, and where activitypattern determiner 266 may evaluate the distribution of the trackedvariables for patterns. In the following example, a tracked variable fora user activity event is a timestamp corresponding to an observedinstance of the user activity event. However, it will be appreciatedthat, conceptually, the following can be applied to different types ofhistorical values for tracked activity features (variables).

A bag of timestamps (i.e., values of a given tracked variable) can bedenoted as {t_(m)}_(m=1) ^(M), and mapped to a two-dimensional histogramof hours and days of the week. The two-dimensional histogram cancomprise a summation over the instances of the user-device interaction,such as:

h _(ij)=Σ_(m=1) ^(M) I[dayOfWeek[t _(m)]=i]∧I[hourOfDay[t _(m)]=j].

This histogram can be used to determine derivative histograms. Forexample, a day of the week histogram may correspond to:h_(j)=Σ_(i)h_(ij). An hour of the day histogram may correspond to:h_(i)=Σ_(j)h_(ij). As further examples, one or more histograms may bedetermined for particular semantic time resolutions in the form of:h_(iC)=Σ_(jϵC)h_(ij). Any of various semantic time resolutions may beemployed, such as weekdays and weekends, or morning, afternoon, andnight. An example of the latter is where Cϵ{morning, afternoon, night},morning={9, 10, 11}, afternoon={12, 13, 14, 15, 16}, and night={21, 22,23, 24}.

An additional data structure utilized in representing an event cancomprise the number of distinct time stamps in every calendar week thathas at least one timestamp therein, which may be represented as:

w _(i) ^(j) =∥{m|t _(m) is within the i-th j week period}∥.

As an example, w₂ ³ can denote the number of distinct timestamps duringthe 2^(nd) three-week period of available timestamps. N^((j)) may beutilized to denote the number of j-week time stamps available in thetracked data; for example, N⁽³⁾ denotes the number of three-week periodsavailable in the timestamps.

Activity pattern determiner 266 (or activity pattern inference engine260) may generate a confidence score that quantifies a level ofcertainty that a particular pattern is formed by the historical valuesin the tracked variable. In the following example, the above principlesare applied utilizing Bayesian statistics. In some implementations, aconfidence score can be generated for a corresponding tracked variablethat is indexed by a temporal interval of varying resolution. Fortimestamps, examples include Tuesday at 9 am, a weekday morning, and aWednesday afternoon. The confidence score may be computed by applying aDirchlet-multinomial model and computing the posterior predictivedistribution of each period histogram. In doing so, a prediction foreach bin in a particular histogram may be given by:

${x_{i} = \frac{\alpha_{0} + h_{i}}{\Sigma_{i}^{K}\left( {\alpha_{0} + h_{i}} \right)}};$

where K denotes the number of bins, α₀ is a parameter encoding thestrength of prior knowledge, and

$i^{*} = {\arg \underset{i}{\max \;}{x_{i}.}}$

Then, the pattern prediction is the bin of the histogram correspondingto i* and its confidence is given by x_(i*). As an example, consider ahistogram in which morning=3, afternoon=4, and evening=3. Using α₀=10,the pattern prediction is afternoon, and the confidence score is

$\frac{{10} + 4}{\left( {{10} + 3} \right) + \left( {{10} + 4} \right) + \left( {{10} + 3} \right)} = {\frac{14}{40} \approx {{0.3}{5.}}}$

In accordance with various implementations, more observations result inan increased confidence score, indicating an increased confidence in theprediction. As an example, consider a histogram in which morning=3000,afternoon=4000, and evening=3000. Using a similar calculation, theconfidence score is

$\frac{4010}{10030} \approx {0.4.}$

Also, in some implementations, a confidence score can be generated for acorresponding tracked variable that is indexed by a period and a numberof timestamps. Examples include 1 visit per week, and 3 visits every 2weeks. Using a Gaussian posterior, a confidence score may be generatedfor a pattern for every period resolution, denoted as j. This may beaccomplished by employing the formula:

${= {{\lambda \left( {\frac{1}{N^{(j)}}{\sum_{i}^{N^{(j)}}w_{i}^{(j)}}} \right)} + {\left( {1 - \lambda} \right)\mu_{0}}}},{where}$$\lambda = {\frac{\sigma_{0}^{2}}{\frac{\sigma^{2}}{N^{(j)}} + \sigma_{0}^{2}}.}$

In the foregoing, σ² is the sample variance, and σ₀ ² and μ₀ areparameters to the formula. A confidence score can be computed by takinga fixed interval around the number of time stamps prediction andcomputing the cumulative density as:

$\left. {{conf}_{j} = {{P\left( {{{x -}} < a} \right)} = {\left( {\left. x \right|,{\overset{\hat{}}{\sigma}}^{(j)}} \right){\overset{\hat{}}{\sigma}}^{(j)}}}} \right),{where}$${\overset{\hat{}}{\sigma}}^{(j)} = {\frac{1}{\frac{N^{(j)}}{\sigma^{2}} + \frac{1}{\sigma_{0}^{2}}}.}$

As an example, consider the following observations: w₁ ⁽¹⁾=10, w₂ ⁽¹⁾=1,w₃ ⁽¹⁾=10, w₄ ⁽¹⁾=0, w₁ ⁽²⁾=11, and w₂ ⁽²⁾=10. N⁽¹⁾=4 and N⁽²⁾=2. Usingμ₀=1 and σ₀ ²=10, μ⁽¹⁾=4.075, and conf₁=0.25. Furthermore, μ⁽²⁾=10.31and conf₂=0.99. In the foregoing example, although fewer time stamps areavailable for two week periods, the reduced variance in the user signalsresults in an increased confidence that a pattern exists.

Having determined that a pattern exists, or that the confidence scorefor a pattern is sufficiently high (e.g., satisfies a threshold value),activity pattern determiner 266 may identify that a plurality of useractivities corresponds to a user activity pattern for the user. As afurther example, activity pattern determiner 266 may determine that auser activity pattern is likely to be followed by a user where one ormore of the confidence scores for one or more tracked variables satisfya threshold value.

In some embodiments, patterns of user activity may be determined bymonitoring one or more activity features, as described previously. Thesemonitored activity features may be determined from the user datadescribed previously as tracked variables or as described in connectionto user-data collection component 210. In some cases, the variables canrepresent context similarities and/or semantic similarities amongmultiple user actions (activity events). In this way, patterns may beidentified by detecting variables or features in common over multipleuser actions. More specifically, features associated with a first useraction may be correlated with features of a second user action todetermine a likely pattern. An identified feature pattern may becomestronger (i.e., more likely or more predictable) the more often the useractivity observations that make up the pattern are repeated. Similarly,specific features can become more strongly associated with a useractivity pattern as they are repeated.

In some embodiments, such as the example embodiment shown in system 200,activity pattern determiner 266 includes one or more pattern basedpredictors 267. Pattern based predictors 267 comprises one or morepredictors for predicting a next or future user action taken by the userbased on patterns, such as patterns of behavior or similarity features.At a high level, a pattern-based predictor 267 receives user activityinformation and/or activity features associated with a user activity anddetermines a prediction of the next action or a future action taken bythe user. In an embodiment, a pattern-based predictor 267 includesfunctionality, as further described below, for performing user activityfiltering, determining an activity score, selecting an activity based onthe score, and determining a particular pattern-based prediction.

In one embodiment, a pattern-based predictor 267 uses featuressimilarity identifier 264 to determine features or patterns in commonbetween historical user activities and a recent user activity. Forexample, similarity of periodic features may be determined, from amongthe set of historical user actions, from those historical actions havinga periodic feature in common with a current or recent user action. Thusfor example, if a recent user action happened on a Monday, on the firstday of the month, on an even week, or on a week day, then determiningperiodic features similarity would comprise identifying those historicaluser actions that have features indicating the user action happened on aMonday, those historical user actions having features corresponding tofirst day of the month (any first day, not just Mondays), or happeningon an even week, or a week day. Likewise, behavior features similaritymay be determined to identify sets of historical user actions having aparticular behavior feature in common with a current or recent useraction. For example, if a recent user action includes visiting anews-related website, then determining behavior features similaritywould comprise identifying those historical user actions that havefeatures indicating the user visited a news-related website.

User activity filtering may use the feature similarity determinations tofilter out historical user actions and retain only those historical useractions that have a particular feature (or features) in common with acurrent or recent user action. Thus, in some embodiments, eachpattern-based predictor 267 may be designed (or tuned) for determining aprediction based on a particular feature (or features); for examplethere might be a subset of pattern-based predictors 267 used fordetermining a prediction when the feature indicates a work day, orweekend, or Monday, or particular app access, or application usageduration, etc. Such a pattern-based predictor 267 may need only thosehistorical user actions corresponding to its prediction model. (In someembodiments, such pattern-based predictors may utilize specificprediction algorithms or models, based on their type of patternprediction (prediction model). These algorithms or models may be storedin with pattern inferences logic 230 in storage 225.)

Accordingly, in some embodiments, for each pattern-based predictor 267,user activity filtering may be carried out to determine a set ofhistorical user actions that are relevant to that particularpattern-based predictor 267; which may include, for example,periodic-feature based predictors, behavior-feature based predictors(which may include behavior sequences or sequences of previous useractions), unique or uncommon behavior features (such as when a userperforms an activity at an unusual time when compared to similarhistorical user activities) or other types of feature-based predictors.(In some embodiments, features similarity identifier 264 may determineuser activity sequence similarity (e.g. the sequence of the last K useractions prior to the current action (or a particular recent action) bydetermining a Levenschtein distance between the historical user actionsequence and recent user action sequences.

Additionally, in embodiments where user activity filtering may beperformed such that each predictor 267 determines a subset of historicaluser actions with features that correspond to its prediction criteria,for each predictor 267, for the subset of historical user actions thatpass the filter, user action scoring may be performed on the subset offeatures. User action scoring generally compares similarities offeatures in the current or recent user action and the subset ofhistorical user actions (which may be considered as a comparison ofcontexts) and scores each user action with respect to the similarity ofits features. In particular, some embodiments score not only thosefeatures used for determining the subset of historical user actions, butall (or a larger number) of features available in the current or recentaction and historical actions for comparison. In some embodiments, aBoolean logic process is used (i.e. the features have to be true or havethe same pattern, and if this is satisfied, then the difference betweenthe particular features is determined). The differences may include, forexample, differences in the time-relate features, usage duration,sequence distances, etc. In an embodiment, these differences aredetermined and put into a sigmoid. Further, in an embodiment, asimilarity threshold is used, which may be pre-determined, tunable, oradaptive, or may be initially set to a value based on a population ofusers or may be based on empirical information learned about theparticular user, for example, and may be adaptive based on the number ofhistorical observations. The similarity threshold may be used todetermine whether a particular historical user action is “similarenough” to the particular current or recent user action so as to beconsidered for determining a prediction. In some embodiments, a vectorrepresenting the similarity differences (or similarity score) may bedetermined; for example, where multiple features are evaluated forsimilarity.

The user actions from the subset of historical user actions that aremost similar (or similar enough) to the particular current or recentuser action may be selected. In some embodiments, the selection processuses a similarity threshold, such as described above, to determine thosehistorical user actions that satisfy the similarity threshold. (Althoughthe term “selection” is used, it is contemplated that the selection isperformed by a computer-related process that does not require a personto perform the selection.) The selected historical user actions comprisea set of “example user actions.”

A prediction of the user's next action (or future action) may beinferred based on (or according to) the historical user actions in theset of example user actions. In an embodiment, the predicted next useraction (or future action) is the next user action with the highestobservations count (i.e. the next user action that is predicted the mostbased on the set of example user actions). Those historical user actionsin the et of example user actions that are consistent with theprediction comprise a “prediction support set.” In some embodiments, aprediction probability corresponding to the prediction may bedetermined; for example, based on a ratio of the size of the predictionsupport set vs. the total number of observations (historical useractions in the subset determined by the user action filtering).Moreover, in some embodiments, the prediction may also compriseadditional information related to the predicted user action(s), such asactivity features which characterize the predicted action(s). By way ofexample and not limitation, if the predicted action is that the userwill email a work colleague, additional activity features may indicatethe specific email recipient, the subject matter of the email, theapproximate time the user will access his email program to compose theemail, or other information related to the predicted action. The relatedinformation may also be determined based on the activity features of theprediction support set observations.

Some embodiments also determine a prediction significance, which may bedetermined based on a confidence interval (e.g. a Binomial confidenceinterval) or other appropriate statistical measure. Still further, insome embodiments, the prediction confidence may be based on theprediction probability and the prediction significance (e.g. the productof the prediction probability and prediction significance). Accordingly,some embodiments of activity pattern determiner 266 provide a predictednext (or future) user action or series of next (or future) actions foreach of the pattern based predictors 267.

Some embodiments determine a specific prediction from the one or morepattern-based predictors 267. In an embodiment, an ensemble process isutilized wherein the one or more pattern-based predictors 267 vote, anda selection is determined based on the ensemble-member predictors.Further, in some embodiments, ensemble-member predictors may be weightedbased on learned information about the user actions. In one embodiment,once each pattern based predictor 267 has provided a prediction, theprediction that has the highest corresponding confidence is determinedas the next (or future) predicted user action, and may be considered apattern-based (or history-based) prediction. In some embodiments, theoutput of activity pattern determiner 266 may be stored as inferred userroutines 248 in user profile 240, and in some embodiments may beprovided to an activity pattern consumer 270.

Continuing with FIG. 2, example system 200 includes one or more activitypattern consumers 270, which comprise applications or services thatconsume activity pattern information to provide improved userexperiences. Examples of activity pattern consumers 270 may include,without limitation, content personalization services, user intentinference services, automatic speech recognition services, device powermanagement services, and semantic understanding services.

In particular, a first example activity pattern consumer 270 comprisescontent personalization services. In one embodiment, a contentpersonalization engine 271 is provided to facilitate providing apersonalized user experience. Thus content personalization engine 271may be considered one example of an application or service (or set ofapplications or services) that may consume information about useractivity patterns, which may include the predictions of future useractions as determined by implementations of the present disclosure.

At a high level, example content personalization engine 271 isresponsible for generating and providing aspects of personalized userexperiences, such as personalized content or tailored delivery ofcontent to a user. The content may be provided to the user as apersonalized notification (such as described in connection withpresentation component 220), may be provided to an application orservice of the user (such as a calendar or scheduling application), ormay be provided as part of an API where it may be consumed by yetanother application or service. In one embodiment, the personalizedcontent includes suggesting that the user perform a relevant activity atthe right time before the user performs the activity manually. Forexample, where an activity pattern indicates the user visits his bank'swebsite near the beginning of the month and enters financial informationinto an Excel file, the user may be provided with a recommendationasking the user, “Would you like to visit your bank website?” Uponresponding affirmatively, a browser instance may be provided that hasautomatically navigated to the user's bank's website. Further, theparticular Excel file may be opened automatically when the useracknowledges the suggestion or when the user navigates to the bank'swebsite. Still further, the user may be provided this content (here, thenotification suggestion) at a convenient time, such as when the user isat his home, on an evening, near the beginning of the month.

In one embodiment, the personalized content may include a notificationwhich comprises a reminder, a recommendation, suggestion, request,communication-related data (e.g. an email, instant message, or callnotification), information relevant to the user in some manner, or othercontent that is provided to the user in a way that is personalized. Forexample, content may be provided at a time when the user would mostlikely desire to receive it, such as a work-related notificationprovided at a time according to a user pattern indicating that the useris about to begin a work-related activity, and not providing the contentat a time when it is likely to be dismissed, ignored, or bothersome.

In another example of providing personalized content, where an activitypattern indicates that a user typically checks online for coupons whenshopping at a particular store, then upon user data indicating that auser has entered the store or is likely going to the store (which may bedetermined based on a predicted future activity), the user may beautomatically provided with the online coupons. Further, the particularcoupons provided may be for items that the user typically buys (or forsimilar items from competitors) based on user activity patterninformation, which may include the user's purchasing habits.

In some embodiments, content personalization engine 271 tailors contentfor a user to provide a personalized user experience. For example,content personalization engine 271 may generate a personalizednotification to be presented to a user, which may be provided topresentation component 220. Alternatively, in other embodiments, contentpersonalization engine 271 generates notification content and makes itavailable to presentation component 220, which determines when and how(i.e., what format) to present the notification based on user data, useractivity pattern information. For example, if a user activity patternindicates the user is likely to be driving to work at a time when it isrelevant to present a particular notification, it may be appropriate toprovide that notification in an audio format, thus personalizing it tothe context of the user.) In some embodiments, other services orapplications operating in conjunction with presentation component 220determine or facilitate determining when and how to present personalizedcontent. The personalized content may be stored in a user profile 240,such as in a personalized content component 249.

Some embodiments of content personalization engine 271 evaluate usercontent to determine how to provide the content in an appropriatemanner. For example, content that is determined to be work related maybe withheld from presentation to a user until user activity patternindicates the user is likely to be conducting work-related activity.

A second example activity pattern consumer 270 comprises a user intentinference service. A user's intent may be inferred based on useractivity pattern information. In particular, whenever a user interactswith a computing device, such as when the user engages in a useractivity, it may be assumed that the user has intent. In one embodiment,a user activity pattern may be analyzed along with sensor data collectedby a user device regarding a particular user interaction, such as a userrequest or commend. The user' intent may be inferred based ondetermining a likely intent that is consistent with a user activitypattern. For example, during a user's lunch hour, the user may performan online search with the intent of locating information about theuser's favorite band, R.E.M., and not information about rapid eyemovement (REM) sleep. A user activity pattern indicating that the usertypically browses music-entertainment related websites over his lunchhour and/or other activity pattern information about the user indicatingthat the user recently listened to music by REM may be used to resolvethe user's intent for the query, namely search results related to themusic group REM and not results related to rapid eye movement. Asanother example, suppose while driving home the user issues a spokenrequest to a user device to “call Pat.” However, the user has more thanone “Pat” on his contact list; for example, the user's brother is namedPat and several of the user's friends are also named Pat. Based on aninferred pattern of user activity indicating that the user typicallycalls his family while driving home, it may be determined that theuser's intent is to call his brother Pat because this intent isconsistent with an activity pattern.

A third example activity pattern consumer 270 comprises automatic speechrecognition (ASR) services. In one embodiment of the disclosure, an ASRservice with improved speech recognition and understanding is provided.The improved ASR service may use information from inferred user activitypatterns to more accurately resolve, disambiguate, or understand userspeech. In particular, a user's speech may be resolved to be consistentwith user activity pattern information. For example, suppose that for aparticular user, a user activity pattern indicates that over lunch theuser typically browses news-related websites. Suppose near 12:00 pm on agiven day that the user speaks a request to a personal assistantservice, such as Cortana®, to go to CNN.com, (e.g., “go to see en en dotcorn”). But the use's request also sounds phonetically similar to go toCMM.com or go to CMN.com. Moreover, the user's spoken request may beeven more difficult to recognize if there is background noise present.However, the information from an inferred user activity patternindicating that the user typically browses news-related websites may beused to more accurately resolve and understand the user speech. Here,the improved ASR service may determine that the user likely said“CNN.com” and not CMN.com or CMM.com because CMN.com and CMM.com are notnews-related websites; CNN.com is a news-related website; and navigatingto CNN.com near 12:00 PM is consistent with the user activity pattern ofbrowsing news-related websites over lunch.

A fourth example activity pattern consumer 270 comprises device powermanagement services. For example, in one embodiment of the disclosure, adevice power management service is provided that regulates powerconsumption of a user device based on user activity pattern information.In particular, based on a pattern of when a user typically charges hismobile device, where it is determined that the time until the typicalcharging time is great enough such that, based on the mean consumptionof battery energy, it is projected that the device will run out ofbattery power before the time that user typically charges the device,then the power management service may implement power-saving measures toreduce battery consumption. For example, the power management servicemay automatically switch the device to a battery-saving mode. Similarly,if it is determined that the user is located at or will be located at alocation where the user has previously charged his user device, whichmay be determined based on location-contextual information associatedwith the user activity related to charging the user device, then theuser may be provided a notification recommending that the user chargehis user device.

A fifth example activity pattern consumer 270 comprises communicationsmanagement services. In particular, network bandwidth and othercommunication-related resources may be more efficiently utilizedaccording to information derived from user activity patterns. Forexample, suppose a user has a pattern of watching a streaming ordownloaded a high-definition movie (e.g. a 4k movie) on Friday nights.One embodiment of a communications management service may cache orbuffer the movie in advance of watching it, so that it is available towatch without interruption. (This may be particularly useful for a userwho shares network bandwidth with other users who are watching streamingmovies at the same time thereby reducing available network bandwidth.)In some embodiments, the particular movie (or several movies likely tobe watched by the user), may be determined from a user's watch list, ormay be inferred based on information derived from monitored useractivity, such as user tastes, internet searches, movies watched by theuser's social media friends, etc. As described previously, someembodiments may use contextual information, such as a user's calendarinformation, to determine the likelihood that the use will follow thepattern. Thus where a user's calendar indicates that an event isscheduled for next Friday night, then it may be determined that, becausethe user is not likely to watch a movie, the movie does not need to bedownloaded prior to next Friday night. Further, in some embodiments,where a user has a data cap or where network speed is capped orotherwise limited, the movie may be downloaded at an earlier time sothat it is already available to watch on Friday night. Still further, insome embodiments, the user may be prompted in advance of downloading aparticular movie about whether the user may be interested in watchingthe movie.

Other examples of activity pattern consumers 270 may include, withoutlimitation: (a) a recommendation service that suggests new content to auser based on user patterns and contextual information. For example, auser activity pattern indicates that a user listens to music everyFriday night. Contextual information indicates that the user preferscertain bands or styles of music. Accordingly, on a given Friday night,a recommendation is provided to the user to listen to a new artisthaving a style similar to the user's taste. (b) A user has an activitypattern of going to music concerts to see bands that sing songs that theuser frequently enjoys. A personal assistant application servicemonitors local concerts and determines that a band that sings songs theuser listens to is coming to town. The personal assistant applicationautomatically purchases a ticket for the user when the tickets firstbecome available. Alternatively, the personal assistant service checksthe user's calendar to determine that the user is available on the dateof the concert, and then prompts the user, notifying the user about theconcert, and in some embodiments, asking if the user wants to personalassistant service to purchase a ticket. (d) A user has an activitypattern of watching an online movie on Friday nights. A personalassistant service determines that a user reads certain genres of books,based on information about book purchases and/or e-reader activity bythe user. Based on the user's taste in books, a movie may be recommendedto the user that the user likely will enjoy. The recommended movie maybe automatically downloaded in a manner to preserve bandwidth in advanceof Friday night.

Example system 200 also includes a presentation component 220 that isgenerally responsible for presenting content and related information toa user, such as the personalized content from content personalizationengine 271 or content from other activity pattern consumers 270.Presentation component 220 may comprise one or more applications orservices on a user device, across multiple user devices, or in thecloud. For example, in one embodiment, presentation component 220manages the presentation of content to a user across multiple userdevices associated with that user. Based on content logic, devicefeatures, associated logical hubs, inferred logical location of the userand/or other user data, presentation component 220 may determine onwhich user device(s) content is presented, as well as the context of thepresentation, such as how (or in what format and how much content, whichcan be dependent on the user device or context) it is presented, when itis presented, etc. In particular, in some embodiments, presentationcomponent 220 applies content logic to device features, associatedlogical hubs, inferred logical locations, or sensed user data todetermine aspects of content presentation.

In some embodiments, presentation component 220 generates user interfacefeatures associated with the personalized content. Such features caninclude interface elements (such as graphics buttons, sliders, menus,audio prompts, alerts, alarms, vibrations, pop-up windows,notification-bar or status-bar items, in-app notifications, or othersimilar features for interfacing with a user), queries, and prompts.

As described previously, in some embodiments, a personal assistantservice or application operating in conjunction with presentationcomponent 220 determines when and how (e.g. presenting when the user isdetermined to be at a specific logical location) to present the content.In such embodiments, the content, including content logic, may beunderstood as a recommendation to the presentation component 220 (and/orpersonal assistant service or application) for when and how to presentthe notification, which may be overridden by the personal assistant appor presentation component 220.

Example system 200 also includes storage 225. Storage 225 generallystores information including data, computer instructions (e.g., softwareprogram instructions, routines, or services), logic, profiles, and/ormodels used in embodiments described herein. In an embodiment, storage225 comprises a data store (or computer data memory). Further, althoughdepicted as a single data store component, storage 225 may be embodiedas one or more data stores or may be in the cloud.

As shown in example system 200, storage 225 includes activity patterninferences logic 230, as described previously, and user profiles 240.One example embodiment of a user profile 240 is illustratively providedin FIG. 2. Example user profile 240 includes information associated witha particular user such as user activity information 242, informationabout user accounts and devices 244, user preferences 246, inferred userroutines 248, and personalized content 249. The information stored inuser profile 240 may be available to the activity pattern inferenceengine 260 other components of example system 200.

As described previously, user activity information 242 generallyincludes user information about user actions or activity events, relatedcontextual information, activity features, or other informationdetermined via user activity monitor 280, and may include historical orcurrent user activity information. User accounts and devices 244generally includes information about user devices accessed, used, orotherwise associated with a the user, and/or information related to useraccounts associated with the user; for example, online or cloud-basedaccounts (e.g. email, social media) such as a Microsoft® Net passport,other accounts such as entertainment or gaming-related accounts (e.g.Xbox live, Netflix, online game subscription accounts, etc.) user datarelating to such accounts such as user emails, texts, instant messages,calls, other communications, and other content; social network accountsand data, such as news feeds; online activity; and calendars,appointments, application data, other user accounts, or the like. Someembodiments of user accounts and devices 244 may store informationacross one or more databases, knowledge graphs, or data structures. Asdescribed previously, the information stored in user accounts anddevices 244 may be determined from user-data collection component 210 oruser activity monitor 280 (including one its subcomponents).

User preferences 246 generally include user settings or preferencesassociated with user activity monitoring. By way of example and notlimitation, such settings may include user preferences about specificactivities (and related information) that the user desires be explicitlymonitored or not monitored or categories of activities to be monitoredor not monitored, crowdsourcing preferences, such as whether to usecrowd sourced information, or whether the user's activity patterninformation may be shared as crowdsourcing data; preferences about whichactivity pattern consumers may consumer the user's activity patterninformation; thresholds, and/or notification preferences, as describedherein. As described previously, inferred user routines 248 may includeone or more user pattern(s) determined by activity pattern determiner266, and may also include confidence scores associated with the patternsand/or information related to the activity patterns, such as contextualinformation or semantic information. Personalized content 249 includespersonalized content determined from content personalization engine 271,such as pending or scheduled notifications.

With reference now to FIG. 3, aspects of an example system fordetermining user activity patterns is provided and referenced generallyas system 300. Example system 300 may determine user activity patternsbased on browse activity and application activity from across multipledevices. As shown, example system 300 comprises one or more clientdevices 302, an inferences system 350, and one or more activity patternconsumers 370. In various embodiments, a client devices 302 may beembodied as a user device such as user device 102 a described in FIG. 1.Client device 302 includes an app activity logging pipeline 383 and abrowse activity logging pipeline 385. These logging pipelines may beembodied as client-side applications or services that run on each userdevice associated with a user, and in some embodiments may run inconjunction with applications or inside (or as a part of) applications,such as within a browser or as a browser plug-in or extension. Appactivity logging pipeline 383, in general, manages logging of a user'sapplication (or app) activity, such as application download, launch,access, use (which may include duration), file access via theapplication, and in-application user activity (which may includeapplication content). Browse activity logging pipeline 385, in general,manages logging of a user's browse activity, such as websites visited,social media activity (which may include browse-type activity conductedvia specific browsers or apps like the Facebook® app, Twitter® app,Instagram® app, Pinterest® app, etc.) content downloaded, filesaccessed, and other browse-related user activity. In some embodiments,app activity logging pipeline 383 and a browse activity logging pipeline385 further include functionality of the embodiments described inconnection with app activity logging pipeline 283 and a browse activitylogging pipeline 285 of system 200 in FIG. 2.

As shown in system 200, inferences system 350 comprises activitypatterns inference engine 360, which receives app activity and browseactivity from one or more client devices 302 and uses that informationto infer activity patterns. Activity patterns inference engine 360 maybe embodied as an embodiment of activity patterns inference engine 260,described in connection to FIG. 2. Further, some embodiments ofinferences system 350 may also include other inference engines (notshown). For example, other inference engines might include componentsfor determining venue visiting inferences (i.e., inferring future venuevisits or patterns of venue visits); location inferences (i.e.,inferring location for the user when explicit location information isnot available); or shadow calendar, which may include inferences about auser's availability based on explicit and inferred scheduled events.

Inferences system 350 may provide inferred user activity patterninformation (including predicted future actions) to activity patternconsumers 370. In some instances, an activity pattern consumers 370 maybe embodied as an activity pattern consumer 270, described in connectionto FIG. 2. A further example activity pattern consumers 370 of system300 includes browser A 372 (or a service associated with browser A) thatnavigates to a website according to a predicted user activity pattern;for example, automatically navigating to the user's bank website nearthe beginning of the month, where the user has a pattern of visiting thewebsite at the beginning of the month. Yet another example activitypattern consumers includes app B 373, which may launch and/or load afile, or perform some functionality (e.g. play a song, compose an email,etc.) according to a predicted user activity pattern.

Turning to FIG. 4, a flow diagram is provided illustrating one examplemethod 400 for inferring a user activity pattern. Each block or step ofmethod 400 and other methods described herein comprises a computingprocess that may be performed using any combination of hardware,firmware, and/or software. For instance, various functions may becarried out by a processor executing instructions stored in memory. Themethods may also be embodied as computer-usable instructions stored oncomputer storage media. The methods may be provided by a stand-aloneapplication, a service or hosted service (stand-alone or in combinationwith another hosted service), or a plug-in to another product, to name afew. Accordingly, method 400 may be performed by one or more computingdevices, such as a smart phone or other user device, a server, or by adistributed computing platform, such as in the cloud. The activitypattern may be inferred through the analysis of signal data (or userdata) gathered from one or more user devices associated with the user.

At step 410, identify a set of user devices associated with a user.Embodiments of step 410 may determine a set of user devices based onbased on monitoring user data for user-device related information. Insome embodiments, the set of user devices identified in step 410comprises one user device or a plurality of user devices, such as userdevices 102 a through 102 n, described in connection to FIG. 1.

In one embodiment, information about user devices associated with a usermay be determined from the user data made available via user-datacollection component 210, such as described in connection to useractivity monitor 280 of FIG. 2. For example, as described previously,information about a user device may be sensed or otherwise detected fromuser data, such as my one or more sensors associated with a user device,or may be determined by detecting and analyzing user-device relatedinformation in user data to determine characteristics of the userdevice, such as device hardware, software such as operating system (OS),network-related characteristics, user accounts accessed via the device,and similar characteristics. In one embodiment, the detected userdevices (such as user device 102 a through 102 n) may be polled,interrogated, or otherwise analyzed to determine information about thedevices. In some implementations, this information may be used fordetermining a label or identification of the device (e.g. a device ID)so that the user interaction with one device may be recognized from userinteraction on another device. In some embodiments of step 410, a userdevice may be identified based on user provided information, such as thecase where a user declares or registers the device; for example, bylogging into an account via the user device, installing an applicationon the device, connecting to an online service that interrogates thedevice, or otherwise providing information about the device to anapplication or service.

At step 420, monitor the set of user devices to detect a user activityevent. Embodiments of step 420 monitor user data associated with the setof user devices identified in step 410, to identify or detect a useractivity event (sometimes referred to herein as a user action). In someinstances, an activity event may comprise a series or sequence of userinteractions with one or more user devices.

Some embodiments of step 420 may comprise monitoring sensor data, usinga one or more sensors associated with the set of user devices. Someembodiments of step 420 may use activity event logic to detect the useractivity event, as described in connection to user activity detector282. Further, some implementations of step 420 may be carried out usinga user activity detector component, such as described in system 200 ofFIG. 2. Additional details of embodiments of step 420 are provided inconnection with user activity detector 282 in FIG. 2.

At step 430, determine a set of activity features associated with theactivity event. Upon detecting or otherwise identifying a user activityevent in step 420, embodiments of step 430 determine a set of activityfeatures associated with the detected activity event. Some embodimentsof step 430 determine the set of activity features based at least inpart on sensor data (including data that may be derived from sensor datasuch as interpretive data) provided by one or more sensors associatedwith the set of user devices. In some embodiments, the sensor data maybe provided via a user data collection component as described in FIG. 2.In particular, user data related to the detected activity event andwhich may be determined at least in part from sensor data, and mayinclude interpreted data, contextual data and/or semantic informationrelated to the detected activity event, is received and analyzed todetermine a set of one or more features associated with the useractivity.

As described previously in connection with user activity monitor 280 andactivity features determiner 286 of FIG. 2, activity-related features(or variables) associated with the user activity that may be used foridentifying patterns of user activity. The activity features may bedetermined from information about a user activity, which in someembodiments may include related contextual information about thedetected activity. Additional details of determining contextualinformation related to a detected activity event are described below andin connection to contextual information extractor 284 of FIG. 2.Further, in some embodiments, the contextual information may includesemantic information determined from a semantic analysis performed onthe detected activity event and/or one or more activity featuresassociated with the activity event. For example, while a user-activityfeature may indicate a specific website visited by the user, semanticanalysis may determine the category of website, related websites, themesor topics or other entities associated with the website or useractivity. From the semantic analysis, additional user-activity relatedfeatures semantically related to the user activity may be determined andused for identifying user activity patterns. Additional details ofdetermining activity features that include semantic information relatedto the detected activity event are described in connection to semanticinformation analyzer 262 of FIG. 2.

Examples of activity-related features may include, without limitation,location-related features, such as location of the user device(s) duringthe user activity, prior to and/or after the user activity,venue-related information associated with the location, or otherlocation-related information; time related features, such as time(s) ofday(s), day of week or month the user activity, or the duration of theactivity, or related duration information such as how long the user usedan application associated with the activity; content-related features,such as online activity by the user (e.g. searches, browsed websites,type or category of websites, purchases, social networking activity,communications sent or received including social media posts, which maycomprise online activity occurring prior to or after the detectedactivity event); other features that may be detected concurrent with theuser activity or near the time of the user activity; or any otherfeatures that may be detected or sensed and used for determining apattern of the user activity including semantic features that maycharacterize aspects of the activity, the nature or type of theactivity, and/or user interactions making up the activity. Features mayalso include information about user(s) using the device; and othercontextual features such as user device-related features, usage relatedfeatures; or other information about the user. By way of example and notlimitation, user device-related features may include information aboutdevice type (e.g. desktop, tablet, mobile phone, fitness tracker, heartrate monitor, etc.) hardware properties or profiles, OS or firmwareproperties, device IDs or model numbers, network-related information(e.g. mac address, network name, IP address, domain, work group,information about other devices detected on the local network, routerinformation, proxy or VPN information, other network connectioninformation, etc.), position/motion/orientation related informationabout the user device, power information such as battery level, time ofconnecting/disconnecting a charger, user-access/touch information; usagerelated features may include information about file(s) accessed, appusage (which may also include application data, in-app usage,concurrently running applications), network usage information, useraccount(s) accessed or otherwise used, (such as device account(s), OSlevel account(s), or online/cloud-services related account(s) activity,such as Microsoft® account or Net Passport, online storage account(s),email, calendar, or social networking accounts, etc.; other informationidentifying a user may include login password, biometric data, which maybe provided by a fitness tracker or biometric scanner; and/orcharacteristics of the user(s) who use the device, which may be usefulfor distinguishing users on devices that are shared by more than oneuser.

In some embodiments of step 430, user activity event logic (described inconnection to user activity detector 282) may be utilized to identifyspecific features associated with the detected user activity event. Someimplementations of step 430 may be carried out using an activityfeatures determiner component, such as described in system 200 of FIG.2. Additional details of embodiments of step 430 are provided inconnection with activity features determiner component 286, in FIG. 2.

At step 440, a record of the detected activity event and the activityfeatures associated with the event are stored in an activity event datastore. The activity event data store may comprise a plurality of recordsabout activity events, wherein each record may include information abouta particular activity event including one or more activity featuresassociated with the activity event. In some embodiments, the pluralityof records in the activity event data store comprise records of otheractivity events determined according to steps 410 through 430 of method400. In some instances, some of the other records may includeinformation about activity events (including associated activityfeatures) derived from other users determined to be similar to theparticular user, as described previously. In an embodiment, the activityevent data store comprises a user profile, and may be stored in a useractivity information component, such as user activity informationcomponent 242 of user profile 240 described in connection to FIG. 2.

At step 450, an activity pattern inference engine is used to identify anactivity pattern. In embodiments of step 450, the activity pattern maybedetected based on an analysis of at least a portion of the plurality ofactivity event records to identify a set of activity events havingsimilar activity features. Thus embodiments of step 450 may comprisedetermining a set of user activity events that have in-common features,which may be determined using a features similarity identifiercomponent, such as features similarity identifier 264, described insystem 200 of FIG. 2.

In embodiments, the set of activity events having similar activityfeatures may comprise two or more activity events that form the basis ofthe pattern, as described previously in connection with activity patterndeterminer 266. In some embodiments, the similar activity features maybe in common with two or more of the activity events, may besubstantially similar, or may be similar enough according to asimilarity threshold, as described previously. Further, some embodimentsmay determine a corresponding confidence score associated with theactivity pattern. In some instances, the confidence score may indicate alikelihood that the user will behave according to the pattern, and maybe used for determining whether to provide activity pattern informationto an activity pattern consumer service, such as described in connectionto FIG. 2. In some embodiments, one or more pattern based predictors maybe utilized, such as described in in connection to FIG. 2.

Step 450 may be carried out by an activity pattern inference engine,such as activity pattern inference engine 260 described in connection tosystem 200 of FIG. 2, or one of its subcomponents. Additional details ofembodiments of step 450 are provided in connection with activity patterninference engine 260, in FIG. 2.

At step 460, the activity pattern is stored in an activity pattern datastore. In an embodiment, the activity pattern data store comprises auser profile, and may be stored in a inferred user routines component,such as inferred user routines 248 of user profile 240 described inconnection to FIG. 2. The activity pattern may be accessed by one ormore computing applications that provide enhanced or improved computingexperiences for the user, such as providing personalization services(e.g. personalized content or tailoring the delivery of content to theuser, as described herein), improved speech recognition, more efficientconsumption or bandwidth or power, or improved semantic understanding ofthe user, which may be used for performing disambiguation or otheraspects of understanding input from the user.

In some embodiments, once an activity pattern is determined, it can beused to determine that a probable future activity event will occur at afuture time that is a threshold time from a present time (e.g. at orwithin three hours, twelve hours, one day, five days, two weeks, etc.from the current time) by analyzing the exercise pattern and/or currentor recent information about user activity. The probable future activityevent can be associated with a location, time, condition and/orsituation (e.g., the future activity likely occurs following a certaintype of event, such as after the user performs another activity) andother contextual data than can be used to facilitate an improved userexperience. Further, as described previously, in some embodiments, theuser activity pattern, or an inferred user intent or predictions offuture activity determined therefrom, may be made available to one ormore applications and services that consume this information and providean improved user experience, such as activity pattern consumers 270,271, 372 or 373, described in connection to FIGS. 2 and 3. For example,in an embodiment a modified user experience is provided, which mayinclude presenting (or withholding, or delaying) content to the user ina manner that is personalized, such as described in connection tocontent personalization engine 271 of System 200 (FIG. 2).

In some embodiments, a cloud system (such as the cloud system describedabove) and/or a cloud service may be utilized to perform method 400 soas to provide a improved or enhanced user experiences (such aspersonalized content) to multiple services, which may be running on manydifferent user devices. As such, system 200 can save significantprocessing, bandwidth, storage, and computing resources by centralizingcertain functionality. For example, user-data collection component 210(FIG. 2) can accumulate user data and interpretive data for multipleusers or user devices, such that each user device does not requireseparate and redundant data collection and storage. Additionally, theprocessing and storage of user profile data can be made more secure bybeing disassociated from the user device, which is closely tied to theuser.

With reference now to FIG. 5, a flow diagram is provided illustratingone example method 500 for determining a probable future user actionbased on a pattern of user activity is provided. At step 510 an inferredactivity pattern for a user is accessed. The inferred exercise patternmay be determined as described previously with reference to FIG. 2, 3 or4. In an embodiment, the activity pattern may be accessed from anactivity pattern data store, such as inferred user routines 248 of userprofile 240, described in connection to FIG. 2. As mentioned, a user maybe associated with multiple activity patterns. A particular inferredpattern can be associated with a confidence level or confidence scorethat indicates how likely to pattern is to be followed (i.e. how likelyit is that a prediction based on the pattern is correct).

At step 520, a probable future activity event is determined based on theactivity pattern. Embodiments of step 530 predict a future activityevent, which may include one or a series (or sequence) of future userinteractions likely to occur or likely to be desired by the user totranspire. In some embodiments, step 520 may also determine a confidencescore associated with the determined probable future activity indicatingthe likelihood that the activity event will occur or be desired by theuser to occur. In some embodiments, the probable future activity maycomprise a context, which may be defined by a location, time, userbehavior, or other condition, and may be determined based at least inpart on sensor data.

For example, in some instances, the probable future activity event isdetermined looking at a periodic or behavior context, which may bedefined according to periodic features or behavioral features, asdescribed previously. Thus a periodic context defines when an activityevent associated with activity pattern is likely to occur. For example,if the pattern indicates a user performs a particular activity everyMonday, Wednesday, and Friday then a future activity event could bedetermined for any Monday, Wednesday, or Friday. Similarly, a behavioralcontext defines contextual features present when an activity pattern islikely to occur. For example, if the pattern indicates a user performs aparticular activity when visiting a certain venue, following anotheractivity, or upon another condition manifesting (e.g., a user checks foronline coupons every time he visits a particular store). The behavioralcontext may also be defined in the negative to identify contextualfeatures that define exceptions to the periodic context. For example,the user may not perform certain activities on holidays or when the useris not in a hub or frequently visited venue, such as when the user istraveling or on vacation. In some embodiments wherein a confidence scoreis determined for the probably future activity, the confidence score maybe further determined based on the context.

In some embodiments, the context may be determined based in part on acurrent or recently occurring activity event. In particular, in someembodiments of step 520, the probable future activity may also bedetermined based on a current or recently occurring user activity event.Some embodiments of step 520 may monitor one or more user devicesassociated with a user to detect a current or recently occurring useractivity event. The detected activity event may be used for determininga particular user activity pattern relevant to a context (e.g., theuser's current situation) and/or a next or future activity event in thepattern. Accordingly, the recency of the activity event detected in suchembodiments of step 520 may be based on a length of time such that acontext (e.g., the user's current situation) is still relevant to theactivity pattern. Thus the recency may vary based on the particularactivity pattern.

In some embodiments, the current or recently occurring activity event isdetermined based on sensor data from one or more user devices. Forexample, some embodiments of step 520 may comprise monitoring sensordata, using a one or more sensors associated with the set of userdevices. Some embodiments of step 520 may be performed as described instep 420 of method 400, or may be carried out using user activitydetector 282, described in system 200 of FIG. 2.

At step 530, based on the determined probable future activity, anenhanced user experience may be provided to the user. Embodiments ofstep 530 provide a enhanced user experience, which may be in the form ofa tailored, modified, or personalized service for the user that isdetermined based at least in part on the probably future activitydetermined in step 520, and may further be determined based on anassociated confidence score. Embodiments of step 530 may compriseproviding a service to the user by one or more activity patternconsumers 270, described in connection to FIG. 2. For example, in anembodiment, the enhanced user experience comprises one of arecommendation, notification, request, or suggestion related to theprobably future activity. In one embodiment, it comprises automaticallycarrying out actions related to the probable future activity at a timeor location consistent with the inferred activity pattern. For example,where the user typically browses her bank account near the beginning ofthe month and copies financial information from the bank website into aspreadsheet, an embodiment of step 530 may automatically perform theoperations of launching a browser, navigating to the bank website, andloading the spreadsheet file in a spreadsheet application. In someembodiments, the automated operations may be performed at an appropriatetime, based on the context, such as while the user is at home using thecomputing device (user device) that she typically uses to access thebank website, or not while the user is driving and thus unable tobenefit from the personalized user experience. Additional examples ofenhanced user experiences are described herein and include, by way ofexample and not limitation, personalization services (e.g. providingpersonalized content or tailoring the delivery of content to the user,as described herein), improved speech recognition, more efficientconsumption or bandwidth or power, or improved semantic understanding ofthe user, which may be used for performing disambiguation or otheraspects of understanding input from the user.

Turning now to FIG. 6, a flow diagram is provided illustrating oneexample method 600 for performing disambiguation to determine a userintent is provided.

At step 610, an indication of a user interaction with a user device isreceived. The indication of the user interaction may be based, at leastin part, on sensor data provided by one or more sensors associated withthe user device. The user interaction may be associated with aparticular user activity associated with a user intention (user intent),such as conducting a search query, requesting to initiate a call (oremail, instant message, other communication) with someone, or issuing avoice command, for example.

At step 620, a set of possible user intentions associated with the userinteraction are determined. Embodiments of step 620 identify a set ofpossible user intentions that make it necessary to performdisambiguation in order to determine the user's actual intent. Forexample, where the user interaction is related to a voice command, theset of intents may correspond to different computer-interpretations ofuser's utterance (e.g., whether the spoke “CNN.com” and thus intends tonavigate to the news website, or spoke “CMN.com and intends to navigateto the Consumer Media Network website). Similarly, where the user isperforming a search query, the user's intent for the search results(e.g., where the user's search terms are R.E.M., whether the userintends to search on the music band or REM sleep).

At step 630, an inferred user activity pattern is accessed. Embodimentsof step 630 may access the inferred user activity pattern as describedin step 510 of method 500. Further, the particular inferred useractivity pattern that is accessed may be determined based on a context,such as described in step 520. In particular, the context may bedetermined according to the location, time, user behavior or activityrelated to the user's current situation and/or the indicated userinteraction in step 610. In some embodiments of step 630, a plurality ofuser activity patterns are accessed. Each user activity pattern mayinclude an associated confidence score reflecting a likelihood that thepattern will be followed, as described herein.

At step 640, a probable future activity event is determined based on theactivity pattern determined in step 630. Some embodiments of step 640further determine the probably future activity event based on thecontext described in connection to step 630. Some embodiments of step640 may be performed as described in step 520 of method 500.

At step 650, select the user intent that is most consistent with theprobably future activity. Embodiments of step 650 determine the userintention, from the set of possible user intents determined in step 620,that is most consistent with the probably future activity. Inparticular, consistency may be determined by conducting an analysis todetermine whether carrying out an operation according to each intentionin the set of intents will result in activity having activity featuresin common with the activity pattern accessed in step 630. For example,suppose a user has a habit of browsing calling family members whiledriving home from work, the user has a brother and several friends namedPat, and issues a voice command to “call Pat.” The context indicatesthat the user is driving, has just left his office, and it is the end ofthe workday. Accordingly, the set of intents includes call an intentionto call each of the user's contacts named Pat. But only one of theseintentions—calling the brother, is consistent with the user's activitypattern of calling family members while driving home from work.Therefore, this intention would be selected. In this way, disambiguationis performed.

In some embodiments, the set of possible user intentions determined instep 620 are ranked in step 650 according to consistency with a probablefuture activity. In particular, in some embodiment of method 600, morethan one user activity pattern is accessed, with each pattern having anassociated confidence score, the set of possible user intentions may beranked according to likelihood of being the correct user intention. Thehighest ranked intention then may be selected in step 650 and used instep 660.

At step 660, an activity associated with the user interaction isperformed based on the selected intent. Embodiments of step 660 carryout the user activity associated with the user intention selected instep 650. Thus for example, continuing the example described in step650, the activity comprises calling the user's brother Pat.

Accordingly, we have described various aspects of technology directed tosystems and methods for inferring user intent and predicting future useractivity associated with a user's computing devices, which may be usedfor providing an enhanced user experience. It is understood that variousfeatures, sub-combinations, and modifications of the embodimentsdescribed herein are of utility and may be employed in other embodimentswithout reference to other features or sub-combinations. Moreover, theorder and sequences of steps shown in the example methods 400, 500, and600 are not meant to limit the scope of the present disclosure in anyway, and in fact, the steps may occur in a variety of differentsequences within embodiments hereof. Such variations and combinationsthereof are also contemplated to be within the scope of embodiments ofthis disclosure.

Having described various implementations, an exemplary computingenvironment suitable for implementing embodiments of the disclosure isnow described. With reference to FIG. 9, an exemplary computing deviceis provided and referred to generally as computing device 900. Thecomputing device 900 is but one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of embodiments of the disclosure. Neithershould the computing device 900 be interpreted as having any dependencyor requirement relating to any one or combination of componentsillustrated.

Embodiments of the disclosure may be described in the general context ofcomputer code or machine-useable instructions, includingcomputer-useable or computer-executable instructions, such as programmodules, being executed by a computer or other machine, such as apersonal data assistant, a smartphone, a tablet PC, or other handhelddevice. Generally, program modules, including routines, programs,objects, components, data structures, and the like, refer to code thatperforms particular tasks or implements particular abstract data types.Embodiments of the disclosure may be practiced in a variety of systemconfigurations, including handheld devices, consumer electronics,general-purpose computers, more specialty computing devices, etc.Embodiments of the disclosure may also be practiced in distributedcomputing environments where tasks are performed by remote-processingdevices that are linked through a communications network. In adistributed computing environment, program modules may be located inboth local and remote computer storage media including memory storagedevices.

Turning now to FIG. 7, a flow diagram is provided illustrating oneexample method 700 for performing speech recognition to determine anaction that a user desires to be performed. The user action could be aspoken request or command (e.g., “Cortana, please go to the CNNwebsite,” or “Cortana, text Ed that I'm running late.”), a question orquery (e.g., “Cortana, what restaurants are close to the theater?” or“Cortana how heavy is traffic right now?”), or otherwise correspond to adesire or intent of the user.

At a high level, embodiments of method 700 may recognize and understandan utterance of the user associated with the action. In particular,where an ambiguity may be present based on a user's particularutterance, embodiments of method 700 may resolve the ambiguity in amanner consistent with an activity pattern for the user, and thenidentify an action corresponding to the now understood utterance, andperform the action. Thus, using the examples above, embodiments ofmethod 700 may determine that a user likely uttered “CNN.com” and not“CMN.com” where the user has a pattern of viewing news-related contentat the time of the utterance. Based on this understanding, a personalassistant application or service (or browser) may navigate to theCNN.com website. Similarly, where the user has an activity patternindicating the user is likely meeting a friend named Ed, embodiments ofmethod 700 may determine that the user likely uttered “text Ed” and not“text Ted.” Similarly still, some embodiments of method 700 determinethe action that a user desires to be performed based on speechrecognition of a set of personalized expressions or words. For example,a user uttering “call mom” may be determined to call the particularcontact of the user that the user identifies as “mom,” which may bedetermined based on the contextual information. Further, such nicknamesor personalized terms and expressions may be identified and learned fromthe monitored activity including contextual information extracted aboutthe activity events, such as described in connection to contextualinformation extractor 284 or user activity monitor 280, of FIG. 2.

Accordingly, at step 710, information associated with an utterance by auser is received. The information may be received, based, at least inpart, on sensor data provided by one or more sensors associated with auser device. The utterance may correspond to an action desired by theuser to be carried out via the user device, such as conducting a searchquery, requesting to initiate a call (or email, instant message, othercommunication) with someone, or issuing a voice command, for example.

At step 720, a set of possible words corresponding to the utterance isdetermined. Embodiments of step 720 identify a set of words that werelikely uttered by the user. In some instances, the set may include onlya single word or each member of the set of words may comprise a singleword, a phrase, or a series or words or phrases. Thus the set mayrepresent the set of likely alternative utterances spoken by the user.For instance, using an example from above, the set may include {“textEd”, “text Ted”, “text Red”, “texted”, etc.}. In some embodiments ofstep 720, a context may be determined based on the set of possiblewords. For instance, in the previous example, the context may comprisethe user texting or communicating with another person. In someembodiments, the context may be determined as described in connection tocontextual information extractor 284 in FIG. 2.

At step 730, an inferred user activity pattern is accessed. Embodimentsof step 730 may access the inferred user activity pattern as describedin step 510 of method 500 or 630 of method 600. Further, the particularinferred user activity pattern that is accessed may be determined basedon a context, such as described in step 520. In particular, the contextmay be determined according to the location, time, user behavior oractivity related to the user's current situation and/or the informationassociated with the utterance in step 710.

At step 740, a probable future activity event is determined based on theactivity pattern determined in step 730. Some embodiments of step 740further determine the probably future activity event based on thecontext described in connection to step 730. Some embodiments of step740 may be performed as described in step 520 of method 500.

At step 750, select a subset of words most consistent with the probablefuture activity event. Embodiments of step 750 determine the words mostlikely spoken by the user, from the set of possible spoken wordsdetermined in step 720, most consistent with the probable futureactivity determined in step 750. In particular, consistency may bedetermined by conducting an analysis to determine whether carrying outan operation according to each action associated with the set of wordswill result in an action having activity features in common with theactivity pattern accessed in step 730. For example, suppose a user has ahabit of calling family members while driving home from work, the userhas a brother named Pat and a friend named Matt, and issues a voicecommand to “call Pat.” The context indicates that the user is driving,has just left his office, and it is the end of the workday. Accordingly,the set of words corresponding to the user's utterance includes at least“call Pat” and “call Matt.” But only one of these—calling the brother,is consistent with the user's activity pattern of calling family memberswhile driving home from work. Therefore, this subset of words (i.e.,“call Pat) would be selected. In this way, more accurate resolution ofspeech is performed.

At step 760, an action corresponding to the subset of words isdetermined. Embodiments of step 760 determine an action corresponding tothe subset of words determined in step 750. The action may comprise oneor more services, processes, routines, functions, or similar activityperformed or facilitated by a user device, or initiating a controlsignal to cause a component to carry out the action. For instance, inthe previous example, the action comprises initiating a call to Pat (orinitiating a control signal for a communication component on a phone tocall Pat). Without limitation, other examples of actions determined instep 760 may comprise other communication (e.g. texting, emailing,instant messaging, posting, etc., which may also include composing orcreating the content for the communication, such as drafting an emailmessage) performing a query, launching an application, navigating to awebsite, playing a particular multimedia content, controlling operationof a device, service, routine, or function, performing a command,request, or any other action capable of being performed or facilitatedvia a user device. In some embodiments, of step 760, the action isdetermined from a set or library of actions that the user device iscapable of performing or facilitating.

At step 770, perform the action determined in step 760. Embodiments ofstep 770 carry out an action corresponding to the subset of wordsdetermined in step 750. Thus for example, continuing the exampledescribed in step 750, the action comprises initiating a phone call theuser's brother Pat. Some embodiments of step 770 comprise initiating acontrol signal to facilitate performing the action. For instance, wherea user asks to turn on the lights, or adjust the thermostat to bewarmer, step 770 may initiate a control signal (or request or similarcommunication specifying the action) to a light-controller component orsmart thermostat.

Turning now to FIG. 8, a perspective 801 is illustratively provided thatdepicts an example of a computer-performed speech recognition (orautomatic speech recognition (ASR)) and understanding system accordingto an embodiment of the disclosure. The ASR system shown in FIG. 8 isjust one example of an ASR system that is suitable for use with anembodiment of the disclosure for determining recognized speech. It iscontemplated that other variations of ASR systems or spoken languageunderstanding (SLU) systems (not shown) may be used including ASRsystems that include fewer components than the example ASR system shownhere, or additional components not shown in FIG. 8.

Perspective 801 shows a sensor 850 that senses acoustic information(audibly spoken words or speech 890) provided by a user-speaker 895.Sensor 850 may comprise one or more microphones or acoustic sensors,which may be embodied as sensor 103 a or 107, or on a user device, suchas user devices 102 or 104, described in FIG. 1. Sensor 850 converts thespeech 890 and into acoustic signal information 853 that maybe providedto a feature extractor 855 (or may be provided directly to decoder 860,in some embodiments). In some embodiments, the acoustic signal mayundergo pre-processing (not shown) before feature extractor 855. Featureextractor 855 generally performs feature analysis to determine theparameterize useful features of the speech signal while reducing noisecorruption or otherwise discarding redundant or unwanted information.Feature extractor 855 transforms the acoustic signal into a features 858(which may comprise a speech corpus) appropriate for the models used bydecoder 860.

Decoder 860 comprises one or more acoustic models (AM) 865 and languagemodels (LM) 870. In embodiments of the disclosure implemented inconnection to an ASR (or SLU) system, a particular embodiment may beimplemented as a subcomponent of decoder 860, LM 870, AM 865, or as aseparate component (not shown) in the example ASR (or SLU) system.

AM 865 comprises statistical representations of distinct sounds thatmake up a word, which may be assigned a label called a “phenome.” AM 865models the phenomes based on the speech features and provides to LM 870a corpus or set of words, which may comprise a sequence of wordscorresponding to the speech corpus, or in some instances a set ofalternative sequences, each corresponding to possible utterances by theuser-speaker 895. LM 870 receives the corpus of words, and determines arecognized speech 880, which may comprise a subset of words (which maybe a single word, phrase, or plurality of words or phrases), includingone interpretation (or recognition) of the utterance, from a set ofalternative interpretations (or recognitions). In this way, a subset ofwords representing a likely utterance spoken by a user may bedetermined.

Accordingly, we have described various aspects of technology directed tosystems and methods for inferring user intent and predicting future useractivity associated with a user's computing devices, which may be usedfor providing an enhanced user experience. It is understood that variousfeatures, sub-combinations, and modifications of the embodimentsdescribed herein are of utility and may be employed in other embodimentswithout reference to other features or sub-combinations. Moreover, theorder and sequences of steps shown in the example methods 400, 500, 600,and 700 are not meant to limit the scope of the present disclosure inany way, and in fact, the steps may occur in a variety of differentsequences within embodiments hereof. Such variations and combinationsthereof are also contemplated to be within the scope of embodiments ofthis disclosure.

Having described various implementations, an exemplary computingenvironment suitable for implementing embodiments of the disclosure isnow described. With reference to FIG. 9, an exemplary computing deviceis provided and referred to generally as computing device 900. Thecomputing device 900 is but one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of embodiments of the disclosure. Neithershould the computing device 900 be interpreted as having any dependencyor requirement relating to any one or combination of componentsillustrated.

Embodiments of the disclosure may be described in the general context ofcomputer code or machine-useable instructions, includingcomputer-useable or computer-executable instructions, such as programmodules, being executed by a computer or other machine, such as apersonal data assistant, a smartphone, a tablet PC, or other handhelddevice. Generally, program modules, including routines, programs,objects, components, data structures, and the like, refer to code thatperforms particular tasks or implements particular abstract data types.Embodiments of the disclosure may be practiced in a variety of systemconfigurations, including handheld devices, consumer electronics,general-purpose computers, more specialty computing devices, etc.Embodiments of the disclosure may also be practiced in distributedcomputing environments where tasks are performed by remote-processingdevices that are linked through a communications network. In adistributed computing environment, program modules may be located inboth local and remote computer storage media including memory storagedevices.

With reference to FIG. 9, computing device 900 includes a bus 910 thatdirectly or indirectly couples the following devices: memory 912, one ormore processors 914, one or more presentation components 916, one ormore input/output (I/O) ports 918, one or more I/O components 920, andan illustrative power supply 922. Bus 910 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 9 are shown with lines for the sakeof clarity, in reality, these blocks represent logical, not necessarilyactual, components. For example, one may consider a presentationcomponent such as a display device to be an I/O component. Also,processors have memory. The inventors hereof recognize that such is thenature of the art and reiterate that the diagram of FIG. 9 is merelyillustrative of an exemplary computing device that can be used inconnection with one or more embodiments of the present disclosure.Distinction is not made between such categories as “workstation,”“server,” “laptop,” “handheld device,” etc., as all are contemplatedwithin the scope of FIG. 9 and with reference to “computing device.”

Computing device 900 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 900 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules, orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVDs) or other optical disk storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by computing device 900.Computer storage media does not comprise signals per se. Communicationmedia typically embodies computer-readable instructions, datastructures, program modules, or other data in a modulated data signalsuch as a carrier wave or other transport mechanism and includes anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media, such as awired network or direct-wired connection, and wireless media, such asacoustic, RF, infrared, and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 912 includes computer storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 900includes one or more processors 914 that read data from various entitiessuch as memory 912 or I/O components 920. Presentation component(s) 916presents data indications to a user or other device. In someimplementations presentation component 220 of system 200 may be embodiedas a presentation component 916. Other examples of presentationcomponents may include a display device, speaker, printing component,vibrating component, and the like.

The I/O ports 918 allow computing device 900 to be logically coupled toother devices, including I/O components 920, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc. The I/Ocomponents 920 may provide a natural user interface (NUI) that processesair gestures, voice, or other physiological inputs generated by a user.In some instances, inputs may be transmitted to an appropriate networkelement for further processing. An NUI may implement any combination ofspeech recognition, touch and stylus recognition, facial recognition,biometric recognition, gesture recognition both on screen and adjacentto the screen, air gestures, head and eye tracking, and touchrecognition associated with displays on the computing device 900. Thecomputing device 900 may be equipped with depth cameras, such asstereoscopic camera systems, infrared camera systems, RGB camerasystems, and combinations of these, for gesture detection andrecognition. Additionally, the computing device 900 may be equipped withaccelerometers or gyroscopes that enable detection of motion. The outputof the accelerometers or gyroscopes may be provided to the display ofthe computing device 900 to render immersive augmented reality orvirtual reality.

Some embodiments of computing device 900 may include one or moreradio(s) 924 (or similar wireless communication components). The radio924 transmits and receives radio or wireless communications. Thecomputing device 900 may be a wireless terminal adapted to receivecommunications and media over various wireless networks. Computingdevice 900 may communicate via wireless protocols, such as code divisionmultiple access (“CDMA”), global system for mobiles (“GSM”), or timedivision multiple access (“TDMA”), as well as others, to communicatewith other devices. The radio communications may be a short-rangeconnection, a long-range connection, or a combination of both ashort-range and a long-range wireless telecommunications connection.When we refer to “short” and “long” types of connections, we do not meanto refer to the spatial relation between two devices. Instead, we aregenerally referring to short range and long range as differentcategories, or types, of connections (i.e., a primary connection and asecondary connection). A short-range connection may include, by way ofexample and not limitation, a Wi-Fi® connection to a device (e.g.,mobile hotspot) that provides access to a wireless communicationsnetwork, such as a WLAN connection using the 802.11 protocol; aBluetooth connection to another computing device is a second example ofa short-range connection, or a near-field communication connection. Along-range connection may include a connection using, by way of exampleand not limitation, one or more of CDMA, GPRS, GSM, TDMA, and 802.16protocols.

Many different arrangements of the various components depicted, as wellas components not shown, are possible without departing from the scopeof the claims below. Embodiments of the disclosure have been describedwith the intent to be illustrative rather than restrictive. Alternativeembodiments will become apparent to readers of this disclosure after andbecause of reading it. Alternative means of implementing theaforementioned can be completed without departing from the scope of theclaims below. Certain features and sub-combinations are of utility andmay be employed without reference to other features and sub-combinationsand are contemplated within the scope of the claims.

Accordingly, in one aspect, an embodiment of the present disclosure isdirected to a computerized system comprising one or more sensorsconfigured to provide sensor data; a user activity monitor configured toidentify and monitor a user device and user activity associated with theuser device; an activity pattern inference engine configured todetermine an activity pattern from a plurality of activity events; oneor more processors; and computer storage memory havingcomputer-executable instructions stored thereon which, when executed bythe processor, implement a method of inferring a user activity pattern.The method includes (a) identifying a set of user devices associatedwith a user; (b) monitoring, using the user activity monitor, the set ofuser devices to detect a user activity event; (c) upon detecting a useractivity event, determining a set of activity features associated withthe activity event, the set of activity features determined based atleast in part on the sensor data; (d) storing a record of the activityevent and associated activity features in an activity event data storethat comprises records of a plurality of activity events; (e) using theactivity pattern inference engine to identify an activity pattern basedon an analysis of the plurality of activity events to determine a set ofactivity events having similar activity features; and (f) storing adescription of the activity pattern in an activity pattern data store.

In some embodiments of this system, the activity event comprises one ormore of browsing to a website, launching an application, initiating acommunication, performing a search query, setting a reminder, orscheduling an event on a calendar and the set of associated activityfeatures comprise one or more features related to content associatedwith the activity event (a content feature), date or time of theactivity event (date-time feature), location of the activity event(location feature), device-usage related features associated with theactivity event (device-usage feature), or contextual informationassociated with the activity event. Further, in some embodiments, theactivity features are determined to be similar based on a comparison ofthe same category of activity features using a similarity threshold thatis pre-determined according to the category of the compared activityfeatures, the categories comprising, by way of example and notlimitation, content features, date-time features, location features,device-usage features, or other types of activity features describedherein.

In another aspect, an embodiment of the present disclosure is directedto a computerized system comprising one or more sensors configured toprovide sensor data; one or more processors; and computer storage memoryhaving computer-executable instructions stored thereon which, whenexecuted by the processor, implement a method of inferring a probablefuture user action. The method includes: (a) accessing an inferred useractivity pattern for a user; (b) predicting a probable future activityevent based on the activity pattern and a context determined at least inpart from the sensor data; and (c) providing an enhanced user experiencebased on the determined probable future activity event.

In one embodiment of this system, the provided enhanced user experiencecomprises one of a recommendation, notification, request, or suggestionrelated to the probably future activity. In another embodiment, theprovided enhanced user experience comprises automatically carrying outthe probable future activity at a time or location consistent with theinferred user activity pattern. In yet another embodiment, the predictedprobable future activity event comprises connecting, at a future time, auser device having a battery to a power source to charge the battery,and the enhanced user experience comprises a battery power managementservice that reduces power consumption by the user device based on thefuture time.

In still another embodiment of this system, the provided enhanced userexperience comprises a speech recognition service, and at least one ofthe one or more sensors comprises an acoustic sensor on a user deviceconfigured to convert speech into acoustic information. The method forthe speech recognition service embodiment further comprises (i)receiving acoustic information corresponding to a spoken interactionwith the user device by a user; (ii) based on an analysis of theacoustic information, determining a plurality of word sequencescorresponding to the spoken interaction; (iii) selecting the wordsequence that is most consistent with the predicted probable futureactivity event; and (iv) providing the selected word sequence asrecognized speech corresponding to the spoken interaction.

In yet another aspect, an embodiment of the disclosure is directed to amethod for performing disambiguation to determine a user intent for auser. The method comprises: (a) receiving an indication of a userinteraction with a user device, the user interaction associated with anactivity performed by the user device, the indication based at least inpart on sensor data from one or more sensors associated with the userdevice; (b) determining a set of possible user intents corresponding tothe user interaction; (c) accessing an inferred activity pattern for theuser; (d) determining a probable future activity event based on theinferred activity pattern; (e) selecting, from the set of possible userintents, the user intent that is most consistent with the probablefuture activity event; and (f) performing the activity associated withthe user interaction based on the selected intent. In some embodimentsof this method, the activity associated with the user interactioncomprises a search query, and further comprising filtering searchresults to be provided to the user based on the selected user intent.

What is claimed is:
 1. A computerized system comprising: one or moreprocessors; and computer storage memory having computer-executableinstructions stored thereon which, when executed by the processor,implement a method of disambiguating an indication of a userinteraction, the method comprising: a) receiving a set of words formingan indication of a user interaction; b) logging user activity in apipeline; c) disambiguating the meaning of a subset of the set of wordsforming the indication of the user interaction based on a sequence ofevents in the pipeline; and d) performing a computerized action based onthe disambiguated meaning.
 2. The computerized system of claim 1,wherein the indication of the user interaction is formed from sensordata containing a user utterance.
 3. The computerized system of claim 1,wherein the sequence of events comprises events with similar timeperiod.
 4. The computerized system of claim 3, wherein the sequence ofevents comprises events with similar location.
 5. The computerizedsystem of claim 3, wherein the sequence of events comprises events thatinvolve similar content.
 6. The computerized system of claim 1, whereinthe subset of words are determined to be consistent with a determinedactivity event.
 7. The computerized system of claim 6, wherein thedetermined activity event comprises one or more of browsing to awebsite, launching an application, initiating a communication,performing a search query, setting a reminder, or scheduling an event ona calendar.
 8. The computerized system of claim 1, wherein the user isprovided an indication of the determined subset of words.
 9. Acomputer-implemented method of disambiguating an indication of a userinteraction, the method comprising: a) receiving a set of words formingan indication of a user interaction; b) logging user activity in a datastore; c) determining a user intent for a subset of the words formingthe user indication based on a sequence of events in the data store; andd) performing a computerized action based on the the user intent. 10.The method of claim 9, wherein the indication of the user interaction isformed from sensor data containing a user utterance.
 11. Thecomputerized of claim 9, wherein the sequence of events comprises eventswith similar time period.
 12. The method of claim 11, wherein thesequence of events comprises events with similar location.
 13. Themethod of claim 11, wherein the sequence of events comprises events thatinvolve similar content.
 14. The method of claim 9, wherein the subsetof words are determined to be consistent with a determined activityevent.
 15. A computerized system comprising: one or more sensorsconfigured to provide sensor data, each sensor associated with a userdevice in a set of user devices associated with a user, at least one ofthe one or more sensors comprising an acoustic sensor configured toconvert speech into acoustic information; one or more processors; andcomputer storage memory having computer-executable instructions storedthereon which, when executed by the processor, implement a method ofdisambiguating an indication of a user interaction, the methodcomprising: a) receiving a set of words forming an indication of a userinteraction, from sensor data corresponding to a user utterance; b)logging user activity in a data store; c) disambiguating the meaning ofa subset of the set of words forming the indication of the userinteraction based on a plurality of events in the data store; and d)performing a computerized action based on the disambiguated meaning. 16.The computerized system of claim 15, wherein the plurality of eventscomprises events with similar time period.
 17. The computerized systemof claim 16, wherein the plurality of events comprises events withsimilar location.
 18. The computerized system of claim 16, wherein theplurality of events comprises events that involve similar content. 19.The computerized system of claim 15, wherein the subset of words aredetermined to be consistent with a determined activity event.
 20. Thecomputerized system of claim 15, wherein the user is provided anindication of the determined subset of words.